Схожи или отличны? Применение Item Response Theory для анализа теста «Синонимы» на выборке взрослых с опытом и без опыта институционализации ¹⁷⁶

В России существует недостаток валидных и стандартизированных инструментов для оценки языкового домена у подростков и взрослых. Для восполнения этого пробела была создана языковая тестовая батарея ARFA-RUS... Методика использовалась в масштабном исследовательском проекте, посвященном долгосрочным эффектам институционального ухода на развитие. человека. В настоящей работе в качестве первого шага валидизации методики мы изучили психометрических свойств субтеста «Синонимы» батареи ARFA-RUS с помощью подхода item response theory (IRT). Результаты IRT-анализа показали надёжность теста для оценки низких и средних уровней способности, однако тест требуется дополнить более сложными заданиями, чтобы охватить больший диапазон способности. Субтест «Синонимы» батареи ARFA-RUS оказался слабо подходящим для группы взрослых с опытом институционализации: в этой группе вычисленная латентная переменная объясняла значительно меньшую долю дисперсии по сравнению с группой взрослых, воспитанных в биологических семьях. На уровне отдельных заданий два вопроса продемонстрировали парадоксальные закономерности: вероятность правильного ответа уменьшалась при увеличении способности. Кроме того, одно задание было исключено из окончательной версии субтеста «Синонимы» из-за его плохого соответствия модели и низкой дискриминативной способности.

Ключевые слова: Item Response Theory, психометрика, дифференцированное функционирование заданий, языковые способности, синонимы, тестирование

Рубрика: Анализ данных

Тип: научная статья

DOI: https://doi.org/10.17759/mda.2020100102

Исследование выполнено при финансовой поддержке Правительства Российской Федерации, грант № 14.Z50.31.0027 «Влияние ранней депривации на био-поведенческие показатели развития ребенка», ведущий ученый – Григоренко Е.Л.

Авторы благодарят Мэй Тэн за помощь в редакции текста публикации.

1. Andringa S. et al. Determinants of Success in Native and Non-Native Listening Comprehension: An Individual Differences Approach // Language Learning. 2012. № SUPPL. 2 (62). C. 49–78.
2. Armstrong R. et al. Change in receptive vocabulary from childhood to adulthood: associated mental health, education and employment outcomes // International Journal of Language and Communication Disorders. 2017.
3. Asimina, Ralli M., Melpomeni S., Alexandra T. Language and Psychosocial Skills of Institutionalized Children in Greece // The Open Family Studies Journal. 2017. № 1 (9). C. 76–87.
4. Baker F.B. The basics of item response theory / F.B. Baker, ERIC, 2001. 186 c.
5. Beck I.L., McKeown M.G. Increasing young low-income children’s oral vocabulary repertoires through rich and focused instruction // Elementary School Journal. 2007. № 3 (107). C. 251–271.
6. Bent T. et al. Individual differences in the perception of regional, nonnative, and disordered speech varieties // The Journal of the Acoustical Society of America. 2016. № 5 (140). C. 3775–3786.
7. Beverly B.L., McGuinness T.M., Blanton D.J. Communication and academic challenges in early adolescence for children who have been adopted from the former Soviet Union // Language, Speech, and Hearing Services in Schools. 2008. № 3 (39). C. 303–313.
8. Biemiller A., Slonim N. Estimating root word vocabulary growth in normative and advantaged populations: Evidence for a common sequence of vocabulary acquisition // Journal of Educational Psychology. 2001.
9. Bock R.D., Aitkin M. Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm // Psychometrika. 1981. № 4 (46). C. 443–459.
10. Carrow-Woolfolk E. Comprehensive Assessment of Spoken Language–Second Edition (CASL-2). Torrance, CA: Western Psychological Services, 2017.
11. Chalmers R.P. Mirt: A multidimensional item response theory package for the R environment // Journal of Statistical Software. 2012. № 6 (48).
12. Chalmers R.P., Counsell A., Flora D.B. It Might Not Make a Big DIF: Improved Differential Test Functioning Statistics That Account for Sampling Variability // Educational and Psychological Measurement. 2016. № 1 (76). C. 114–140.
13. Cunningham A.E., Stanovich K.E. What Reading Does for the Mind // American Educator. 1998. № 1 (22). C. 137–149.
14. Desmarais C. et al. Sentence comprehension in postinstitutionalized school-age children // Journal of Speech, Language, and Hearing Research. 2012. № 1 (55). C. 45–54.
15. Drasgow F., Lissak R.I. Modifi ed parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. // Journal of Applied Psychology. 1983.
16. Dunn D.M. Peabody picture vocabulary test-fi fth edition. Bloomington, MN.: Pearson, 2019.
17. Embretson S.E., Reise S.P. Item response theory for psychologists Mahwah, N.J.: Lawrence Erlbaum Associates Publishers, 2000.
18. Fernald A., Marchman V.A., Weisleder A. SES differences in language processing skill and vocabulary are evident at 18 months // Developmental Science. 2013.
19. Fromkin V. et al. The development of language in genie: a case of language acquisition beyond the “critical period” // Brain and Language. 1974. № 1 (1). C. 81–107.
20. Gindis B. Cognitive, Language, and Educational Issues of Children Adopted from Overseas Orphanages // Journal of Cognitive Education and Psychology. 2005. № 3 (4). C. 291–315.
21. Glennen S., Bright B.J. Five years later: Language in school-age internationally adopted children // Seminars in Speech and Language. 2005. Т. 26. № 1. 86–101 с.
22. Goulden R., Nation P., Read J. How large can a receptive vocabulary be? // Applied Linguistics. 1990. № 4 (11). C. 341–363.
23. Greenough W.T., Black J.E., Wallace C.S. Experience and brain development. // Child development. 1987. № 3 (58). C. 539–559.
24. Hammill D.D. et al. TOAL-4: Test of Adolescent and Adult Language—Fourth Edition // 2007.
25. Hart B., Risley T.R. Meaningful differences in the everyday experience of young American children. / B. Hart, T.R. Risley, Paul H Brookes Publishing, 1995.
26. Hart B., Risley T.R. The early catastrophe: The 30 million word gap by age 3 // American educator. 2003. № 1 (27). C. 4–9.
27. Hemphill L., Tivnan T. The Importance of Early Vocabulary for Literacy Achievement in High-Poverty Schools // Journal of Education for Students Placed at Risk (JESPAR). 2008. № 4 (13). C. 426–451.
28. Hiebert E.H., Kamil M.L. Teaching and learning vocabulary: Bringing research to practice Routledge, 2005. 1–279 с.
29. Horn J.L. A rationale and test for the number of factors in factor analysis // Psychometrika. 1965. № 2 (30). C. 179–185.
30. Hu L.T., Bentler P.M. Cutoff criteria for fi t indexes in covariance structure analysis: Conventional criteria versus new alternatives // Structural Equation Modeling. 1999. № 1 (6). C. 1–55.
31. Ivanova M. et al. AutoRAT at your fi ngertips: Introducing the new Russian Aphasia Test on a tablet // Frontiers in Psychology. 2016. (7).
32. Janse E., Jesse A. Working memory affects older adults’ use of context in spoken-word recognition // Quarterly Journal of Experimental Psychology. 2014. № 9 (67). C. 1842–1862.
33. Johnson M.S. Marginal maximum likelihood estimation of item response models in R // Journal of Statistical Software. 2007. № 10 (20). C. 1–24.
34. Kamata A., Bauer D.J. A note on the relation between factor analytic and item response theory models // Structural Equation Modeling: A Multidisciplinary Journal. 2008. № 1 (15). C. 136–153.
35. Kornilov S.A. et al. Language Outcomes in Adults with a History of Institutionalization: Behavioral and Neurophysiological Characterization // Scientifi c Reports. 2019. № 1 (9). C. 4252.
36. Langer M.M. et al. Item response theory detected differential item functioning between healthy and ill children in quality-of-life measures // Journal of Clinical Epidemiology. 2008. № 3 (61). C. 268–276.
37. Linden W.J. van der Handbook of item response theory CRC Press, 2016. 1–595 с.
38. Loman M.M. et al. Postinstitutionalized children’s development: Growth, cognitive, and language outcomes // Journal of Developmental and Behavioral Pediatrics. 2009. № 5 (30). C. 426–434.
39. Lord F.M. Applications of item response theory to practical testing problems Hillsdale, NJ: Lawrence Erlbaum Associates, 1980. 1–274 с.
40. Magis D. et al. A general framework and an R package for the detection of dichotomous differential item functioning // Behavior Research Methods. 2010. № 3 (42). C. 847–862.
41. Martinková P. et al. Checking equity: Why differential item functioning analysis should be a routine part of developing conceptual assessments // CBE Life Sciences Education. 2017. № 2 (16). C. rm2.
42. Mayberry R.I. Cognitive development in deaf children: The interface of language and perception in neuropsychology // Handbook of Neuropsychology. 2002. № Part II (8). C. 71–107.
43. McDermott J. et al. Psychosocial deprivation, executive functions, and the emergence of socioemotional behavior problems // Frontiers in Human Neuroscience. 2013. Т. 7. 167 с.
44. McGregor K.K. et al. Children with developmental language impairment have vocabulary defi cits characterized by limited breadth and depth // International Journal of Language and Communication Disorders. 2013. № 3 (48). C. 307–319.
45. Merz E.C., McCall R.B., Wright A.J. Attention and language as mediators of academic outcomes following early psychosocial deprivation // International Journal of Behavioral Development. 2013. № 5 (37). C. 451–459.
46. Milton J., Treffers-Daller J. Vocabulary size revisited: The link between vocabulary size and academic achievement // Applied Linguistics Review. 2013.
47. Muhamedrahimov R.J. et al. Structural characteristics of the institutional environment for young children // Psychology in Russia: State of the Art. 2016.
48. Nation K. Lexical learning and lexical processing in children with developmental language impairments // Philosophical Transactions of the Royal Society B: Biological Sciences. 2014. Т. 369. № 1634. 20120387 с.
49. National Reading Panel Teaching children to read: An evidence-based assessment of the scientific c research literature on reading and its implications for reading instruction // NIH Publication No. 00-4769. 2000.
50. Paek I., Cole K. Using R for Item Response Theory Model Applications / I. Paek, K. Cole, Routledge, 2019.
51. Quinn J.M., Wagner R.K. Using Meta-analytic Structural Equation Modeling to Study Developmental Change in Relations Between Language and Literacy // Child Development. 2018. № 6 (89). C. 1956–1969.
52. R Core Team (2019) R: A language and environment for statistical computing. // Accessed 1st April 2019. 2019.
53. Raju N.S. The area between two item characteristic curves // Psychometrika. 1988. № 4 (53). C. 495–502.
54. Reise S.P., Henson J.M. A discussion of modem versus traditional psychometrics as applied to personality assessment scales // Journal of personality assessment. 2003. № 2 (81). C. 93–103.
55. Richland L.E., Burchinal M.R. Early Executive Function Predicts Reasoning Development // Psychological Science. 2012. № 1 (24). C. 87–92.
56. Rizopoulos D. Itm: An R package for latent variable modeling and item response theory analyses // Journal of Statistical Software. 2006. № 5 (17). C. 1–25.
57. Rodríguez-Aranda C., Jakobsen M. Differential contribution of cognitive and psychomotor functions to the age-related slowing of speech production // Journal of the International Neuropsychological Society. 2011. № 5 (17). C. 807–821.
58. Scott K.A., Roberts J.A., Glennen S. How well do children who are internationally adopted acquire language? a meta-analysis // Journal of Speech, Language, and Hearing Research. 2011. № 4 (54). C. 1153–1169.
59. Shao Z. et al. What do verbal fl uency tasks measure? Predictors of verbal fl uency performance in older adults // Frontiers in Psychology. 2014. № JUL (5). C. 772.
60. Stahl S.A., Fairbanks M.M. The Effects of Vocabulary Instruction: A Model-Based Meta-Analysis // Review of Educational Research. 1986.
61. Steinberg L., Thissen D. Item response theory. Oxford library of psychology. / New York, NY, US: Oxford University Press, 2013. 336–373 с.
62. Takane Y., Leeuw J. De On the relationship between item response theory and factor analysis of discretized variables // Psychometrika. 1987. № 3 (52). C. 393–408.
63. Tirella L.G. et al. Time use in Russian Baby Homes // Child: Care, Health and Development. 2008.
64. Unsworth N., Spillers G.J., Brewer G.A. Variation in verbal fl uency: A latent variable analysis of clustering, switching, and overall performance // Quarterly Journal of Experimental Psychology. 2011. № 3 (64). C. 447–466.
65. Wechsler D. Wechsler adult intelligence scale–Fourth Edition (WAIS–IV). San Antonio, TX: Pearson, 2008.
66. Wechsler D. Wechsler intelligence scale for children–Fifth Edition (WISC-V) Bloomington, MN: Pearson, 2014.
67. Widhiarso W., Haryanta Examining method effect of synonym and antonym test in verbal abilities measure // Europe’s Journal of Psychology. 2015.
68. Wiig E.H., Secord W.A., Semel E. Clinical evaluation of language fundamentals: CELF-5 Bloomington, MN: Pearson, 2013.
69. Williams K. Expressive vocabulary test, third edition. Bloomington, MN: Pearson, 2018.
70. Windsor J. et al. Language acquisition with limited input: Romanian institution and foster care // Journal of Speech, Language, and Hearing Research. 2007. № 5 (50). C. 1365–1381.
71. Windsor J. et al. Effect of foster care on young children’s language learning // Child Development. 2011. № 4 (82). C. 1040–1046.
72. Windsor J. et al. Effect of foster care on language learning at eight years: Findings from the Bucharest early intervention project // Journal of Child Language. 2013. № 3 (40). C. 605–627.
73. Yap M.J. et al. Individual differences in visual word recognition: Insights from the English Lexicon Project // Journal of Experimental Psychology: Human Perception and Performance. 2012. № 1 (38). C. 53–79.
74. Zanon C. et al. An application of item response theory to psychological test development // Psicologia: Refl exão e Crítica. 2016. № 1 (29). C. 1–10.
75. Прихода Н.А. Оценка развития русского языка (ОРРЯ) как стандартизованная методика диагностики коммуникативной функции у детей от 3 до 9 лет // Психологическая наука и образование. 2016. № 3 (21). C. 25–33.