Similar or different? An Item Response Theory Analysis of the Synonyms Test in Adults with and without a History of Institutionalization

The lack of valid and standardized instruments, directed on an assessment of the language domain in adolescents and adults in Russia postulates the urgent necessity of their development. To fi ll this gap, the language battery, ARFA-RUS, was created and applied in a large project investigating the long-term consequences of raring in institutional care settings on human development. In the current study, an Item Response Theory (IRT) approach was used to examine the psychometric properties of the Synonyms Subtest of ARFA-RUS as the fi rst step of validation of the battery. IRT results demonstrated the test is reliable for the low-to-moderate levels of the assessed ability; yet, to capture a wider ability range, more diffi cult items are needed. The ARFA-RUS Synonyms Subtest was less suitable for the postinstitutionalized group of adults; in this group, the latent ability estimate explained a lower percentage of variance in comparison to adults raised in biological families. With regard to item-specifi c analyses, two items demonstrated paradoxical patterns with decreased probability of correct response at increased ability. In addition, one item was eliminated from the fi nal version of the Synonyms Subtest due to its poor item fi t and low discrimination value.

Language is a fundamental human ability, and it is a fundamental component of many different skills and processes (i.e., memory, executive functions, learning). Like other complex skills, language development is influenced by a variety of genetic and environmental factors and their interactions.

1. Andringa S. et al. Determinants of Success in Native and Non-Native Listening Comprehension: An Individual Differences Approach // Language Learning. 2012. № SUPPL. 2 (62). C. 49–78.
2. Armstrong R. et al. Change in receptive vocabulary from childhood to adulthood: associated mental health, education and employment outcomes // International Journal of Language and Communication Disorders. 2017.
3. Asimina, Ralli M., Melpomeni S., Alexandra T. Language and Psychosocial Skills of Institutionalized Children in Greece // The Open Family Studies Journal. 2017. № 1 (9). C. 76–87.
4. Baker F.B. The basics of item response theory / F.B. Baker, ERIC, 2001. 186 c.
5. Beck I.L., McKeown M.G. Increasing young low-income children’s oral vocabulary repertoires through rich and focused instruction // Elementary School Journal. 2007. № 3 (107). C. 251–271.
6. Bent T. et al. Individual differences in the perception of regional, nonnative, and disordered speech varieties // The Journal of the Acoustical Society of America. 2016. № 5 (140). C. 3775–3786.
7. Beverly B.L., McGuinness T.M., Blanton D.J. Communication and academic challenges in early adolescence for children who have been adopted from the former Soviet Union // Language, Speech, and Hearing Services in Schools. 2008. № 3 (39). C. 303–313.
8. Biemiller A., Slonim N. Estimating root word vocabulary growth in normative and advantaged populations: Evidence for a common sequence of vocabulary acquisition // Journal of Educational Psychology. 2001.
9. Bock R.D., Aitkin M. Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm // Psychometrika. 1981. № 4 (46). C. 443–459.
10. Carrow-Woolfolk E. Comprehensive Assessment of Spoken Language–Second Edition (CASL-2). Torrance, CA: Western Psychological Services, 2017.
11. Chalmers R.P. Mirt: A multidimensional item response theory package for the R environment // Journal of Statistical Software. 2012. № 6 (48).
12. Chalmers R.P., Counsell A., Flora D.B. It Might Not Make a Big DIF: Improved Differential Test Functioning Statistics That Account for Sampling Variability // Educational and Psychological Measurement. 2016. № 1 (76). C. 114–140.
13. Cunningham A.E., Stanovich K.E. What Reading Does for the Mind // American Educator. 1998. № 1 (22). C. 137–149.
14. Desmarais C. et al. Sentence comprehension in postinstitutionalized school-age children // Journal of Speech, Language, and Hearing Research. 2012. № 1 (55). C. 45–54.
15. Drasgow F., Lissak R.I. Modifi ed parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. // Journal of Applied Psychology. 1983.
16. Dunn D.M. Peabody picture vocabulary test-fi fth edition. Bloomington, MN.: Pearson, 2019.
17. Embretson S.E., Reise S.P. Item response theory for psychologists Mahwah, N.J.: Lawrence Erlbaum Associates Publishers, 2000.
18. Fernald A., Marchman V.A., Weisleder A. SES differences in language processing skill and vocabulary are evident at 18 months // Developmental Science. 2013.
19. Fromkin V. et al. The development of language in genie: a case of language acquisition beyond the “critical period” // Brain and Language. 1974. № 1 (1). C. 81–107.
20. Gindis B. Cognitive, Language, and Educational Issues of Children Adopted from Overseas Orphanages // Journal of Cognitive Education and Psychology. 2005. № 3 (4). C. 291–315.
21. Glennen S., Bright B.J. Five years later: Language in school-age internationally adopted children // Seminars in Speech and Language. 2005. Т. 26. № 1. 86–101 с.
22. Goulden R., Nation P., Read J. How large can a receptive vocabulary be? // Applied Linguistics. 1990. № 4 (11). C. 341–363.
23. Greenough W.T., Black J.E., Wallace C.S. Experience and brain development. // Child development. 1987. № 3 (58). C. 539–559.
24. Hammill D.D. et al. TOAL-4: Test of Adolescent and Adult Language—Fourth Edition // 2007.
25. Hart B., Risley T.R. Meaningful differences in the everyday experience of young American children. / B. Hart, T.R. Risley, Paul H Brookes Publishing, 1995.
26. Hart B., Risley T.R. The early catastrophe: The 30 million word gap by age 3 // American educator. 2003. № 1 (27). C. 4–9.
27. Hemphill L., Tivnan T. The Importance of Early Vocabulary for Literacy Achievement in High-Poverty Schools // Journal of Education for Students Placed at Risk (JESPAR). 2008. № 4 (13). C. 426–451.
28. Hiebert E.H., Kamil M.L. Teaching and learning vocabulary: Bringing research to practice Routledge, 2005. 1–279 с.
29. Horn J.L. A rationale and test for the number of factors in factor analysis // Psychometrika. 1965. № 2 (30). C. 179–185.
30. Hu L.T., Bentler P.M. Cutoff criteria for fi t indexes in covariance structure analysis: Conventional criteria versus new alternatives // Structural Equation Modeling. 1999. № 1 (6). C. 1–55.
31. Ivanova M. et al. AutoRAT at your fi ngertips: Introducing the new Russian Aphasia Test on a tablet // Frontiers in Psychology. 2016. (7).
32. Janse E., Jesse A. Working memory affects older adults’ use of context in spoken-word recognition // Quarterly Journal of Experimental Psychology. 2014. № 9 (67). C. 1842–1862.
33. Johnson M.S. Marginal maximum likelihood estimation of item response models in R // Journal of Statistical Software. 2007. № 10 (20). C. 1–24.
34. Kamata A., Bauer D.J. A note on the relation between factor analytic and item response theory models // Structural Equation Modeling: A Multidisciplinary Journal. 2008. № 1 (15). C. 136–153.
35. Kornilov S.A. et al. Language Outcomes in Adults with a History of Institutionalization: Behavioral and Neurophysiological Characterization // Scientifi c Reports. 2019. № 1 (9). C. 4252.
36. Langer M.M. et al. Item response theory detected differential item functioning between healthy and ill children in quality-of-life measures // Journal of Clinical Epidemiology. 2008. № 3 (61). C. 268–276.
37. Linden W.J. van der Handbook of item response theory CRC Press, 2016. 1–595 с.
38. Loman M.M. et al. Postinstitutionalized children’s development: Growth, cognitive, and language outcomes // Journal of Developmental and Behavioral Pediatrics. 2009. № 5 (30). C. 426–434.
39. Lord F.M. Applications of item response theory to practical testing problems Hillsdale, NJ: Lawrence Erlbaum Associates, 1980. 1–274 с.
40. Magis D. et al. A general framework and an R package for the detection of dichotomous differential item functioning // Behavior Research Methods. 2010. № 3 (42). C. 847–862.
41. Martinková P. et al. Checking equity: Why differential item functioning analysis should be a routine part of developing conceptual assessments // CBE Life Sciences Education. 2017. № 2 (16). C. rm2.
42. Mayberry R.I. Cognitive development in deaf children: The interface of language and perception in neuropsychology // Handbook of Neuropsychology. 2002. № Part II (8). C. 71–107.
43. McDermott J. et al. Psychosocial deprivation, executive functions, and the emergence of socio-emotional behavior problems // Frontiers in Human Neuroscience . 2013. Т. 7. 167 с.
44. McGregor K.K. et al. Children with developmental language impairment have vocabulary deficits characterized by limited breadth and depth // International Journal of Language and Communication Disorders. 2013. № 3 (48). C. 307–319.
45. Merz E.C., McCall R.B., Wright A.J. Attention and language as mediators of academic outcomes following early psychosocial deprivation // International Journal of Behavioral Development. 2013. № 5 (37). C. 451–459.
46. Milton J., Treffers-Daller J. Vocabulary size revisited: The link between vocabulary size and academic achievement // Applied Linguistics Review. 2013.
47. Muhamedrahimov R.J. et al. Structural characteristics of the institutional environment for young children // Psychology in Russia: State of the Art. 2016.
48. Nation K. Lexical learning and lexical processing in children with developmental language impairments // Philosophical Transactions of the Royal Society B: Biological Sciences. 2014. Т. 369. № 1634. 20120387 с.
49. National Reading Panel Teaching children to read: An evidence-based assessment of the scientifi c research literature on reading and its implications for reading instruction // NIH Publication No.00-4769. 2000.
50. Paek I., Cole K. Using R for Item Response Theory Model Applications / I. Paek, K. Cole, Routledge, 2019.
51. Quinn J.M., Wagner R.K. Using Meta-analytic Structural Equation Modeling to Study Developmental Change in Relations Between Language and Literacy // Child Development. 2018. № 6 (89). C. 1956–1969.
52. R Core Team (2019) R: A language and environment for statistical computing. // Accessed 1st April 2019. 2019.
53. Raju N.S. The area between two item characteristic curves // Psychometrika. 1988. № 4 (53). C. 495–502.
54. Reise S.P., Henson J.M. A discussion of modem versus traditional psychometrics as applied to personality assessment scales // Journal of personality assessment. 2003. № 2 (81). C. 93–103.
55. Richland L.E., Burchinal M.R. Early Executive Function Predicts Reasoning Development // Psychological Science. 2012. № 1 (24). C. 87–92.
56. Rizopoulos D. Itm: An R package for latent variable modeling and item response theory analyses // Journal of Statistical Software. 2006. № 5 (17). C. 1–25.
57. Rodríguez-Aranda C., Jakobsen M. Differential contribution of cognitive and psychomotor functions to the age-related slowing of speech production // Journal of the International Neuropsychological Society. 2011. № 5 (17). C. 807–821.
58. Scott K.A., Roberts J.A., Glennen S. How well do children who are internationally adopted acquire language? a meta-analysis // Journal of Speech, Language, and Hearing Research. 2011. № 4 (54). C. 1153–1169.
59. Shao Z. et al. What do verbal fl uency tasks measure? Predictors of verbal fl uency performance in older adults // Frontiers in Psychology. 2014. № JUL (5). C. 772.
60. Stahl S.A., Fairbanks M.M. The Effects of Vocabulary Instruction: A Model-Based Meta-Analysis // Review of Educational Research. 1986.
61. Steinberg L., Thissen D. Item response theory. Oxford library of psychology. / New York, NY, US: Oxford University Press, 2013. 336–373 с.
62. Takane Y., Leeuw J. De On the relationship between item response theory and factor analysis of discretized variables // Psychometrika. 1987. № 3 (52). C. 393–408.
63. Tirella L.G. et al. Time use in Russian Baby Homes // Child: Care, Health and Development. 2008.
64. Unsworth N., Spillers G.J., Brewer G.A. Variation in verbal fl uency: A latent variable analysis of clustering, switching, and overall performance // Quarterly Journal of Experimental Psychology. 2011. № 3 (64). C. 447–466.
65. Wechsler D. Wechsler adult intelligence scale–Fourth Edition (WAIS–IV). San Antonio, TX: Pearson, 2008.
66. Wechsler D. Wechsler intelligence scale for children–Fifth Edition (WISC-V) Bloomington, MN: Pearson, 2014.
67. Widhiarso W. Haryanta Examining method effect of synonym and antonym test in verbal abilities measure // Europe’s Journal of Psychology. 2015.
68. Wiig E.H., Secord W.A., Semel E. Clinical evaluation of language fundamentals: CELF-5 Bloomington, MN: Pearson, 2013.
69. Williams K. Expressive vocabulary test, third edition. Bloomington, MN: Pearson, 2018.
70. Windsor J. et al. Language acquisition with limited input: Romanian institution and foster care // Journal of Speech, Language, and Hearing Research. 2007. № 5 (50). C. 1365–1381.
71. Windsor J. et al. Effect of foster care on young children’s language learning // Child Development. 2011. № 4 (82). C. 1040–1046.
72. Windsor J. et al. Effect of foster care on language learning at eight years: Findings from the Bucharest early intervention project // Journal of Child Language. 2013. № 3 (40). C. 605–627.
73. Yap M.J. et al. Individual differences in visual word recognition: Insights from the English Lexicon Project // Journal of Experimental Psychology: Human Perception and Performance. 2012. № 1 (38). C. 53–79.
74. Zanon C. et al. An application of item response theory to psychological test development // Psicologia: Refl exão e Crítica. 2016. № 1 (29). C. 1–10.
75. Prikhoda N.A. Russian Language Development Assessment as a Standardized Technique for Assessing Communicative Function in Children Aged 3–9 Years. Psikhologicheskaya nauka i obrazovanie = Psychological Science and Education, 2016. Vol. 21, no. 3, pp. 25–33. doi:10.17759/pse.2016210304. (In Russ., аbstr. in Engl.)