Testing the Tests: Investigating Measurement Invariance in Assessments of Cognitive and Language Abilities

This paper discusses the measurement invariance of assessment instruments. The purpose of measurement invariance testing is to assess the extent to which a measured construct has comparable meaning across different groups or measurement points, to ensure that comparisons of these groups or measures to each other are appropriate. The paper provides an overview of recent and classic papers discussing conceptual and methodological aspects of invariance testing. Specific focus is placed on the available data on invariance testing for the assessments of cognitive and language abilities, and on the significance of invariance for the accurate assessment of individual and between-group differences. Finally, current approaches to invariance testing within structural equation modelling and item response theory are discussed. This paper may serve as an introduction to the concept of measurement invariance. It will be useful to readers interested in improving the quality of measurement instruments and making more reliable inferences about psychological phenomena.

1. Schneider L., Strobl C., Zeileis A., Debelak R. An R toolbox for score-based measurement invariance tests in IRT models. Behavior Research Methods, 2021. Vol. 54, pp. 2101—2113. DOI:10.3758/s13428-021-01689-0
2. Bauer D.J., Belzak W.C.M., Cole V.T. Simplifying the Assessment of Measurement Invariance over Multiple Background Variables: Using Regularized Moderated Nonlinear Factor Analysis to Detect Differential Item Functioning. Structural Equation Modeling: A Multidisciplinary Journal, 2020. Vol. 27, no. 1, pp. 43—55. DOI:10.1080/10705511.2019.1642754
3. Boer D., Hanke K., He J. On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests. Journal of Cross-Cultural Psychology, 2018. Vol. 49, no. 5, pp. 713—734. DOI:10.1177/0022022117749042
4. Choi Y.-J., Asilkalkan A.R. Packages for Item Response Theory Analysis: Descriptions and Features. Measurement: Interdisciplinary Research and Perspectives, 2019. Vol. 17, no. 3, pp. 168—175. DOI:10.1080/15366367.2019.1586404
5. Talbott E., De Los Reyes A., Kearns D.M., Mancilla-Martinez J., Wang M. Evidence-Based Assessment in Special Education Research: Advancing the Use of Evidence in Assessment Tools and Empirical Processes. Exceptional Children, 2023. Vol. 89, no. 4, pp. 467—487. DOI:10.1177/00144029231171092
6. Jonson J.L., Geisinger K.F. (eds.), Fairness in educational and psychological testing: examining theoretical, research, practice, and policy implications of the 2014 standards. Washington: American Educational Research Association, 2022. 466 p. DOI:10.2307/j.ctv2kzv0fw
7. Flake J.K., Fried E.I. Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science, 2020. Vol. 3, no. 4, pp. 456—465. DOI:10.1177/2515245920952393
8. Funder D.C., Gardiner G. MIsgivings about measurement invariance. European Journal of Personality, 2024. 7 p. Ahead of Print. DOI:10.1177/08902070241228338
9. Hajovsky D.B., Chesnut S.R. Examination of differential effects of cognitive abilities on reading and mathematics achievement across race and ethnicity: Evidence with the WJ IV. Journal of School Psychology, 2022. Vol. 93, pp. 1—27. DOI:10.1016/j.jsp.2022.05.001
10. Han K., Colarelli S.M., Weed N.C. Methodological and statistical advances in the consideration of cultural diversity in assessment: A critical review of group classification and measurement invariance testing. Psychological Assessment, 2019. Vol. 31(12), pp. 1481—1496. DOI:10.1037/pas0000731
11. Van Der Linden W.J. (ed.), Handbook of Item Response Theory: Three Volume Set. Boca Raton: CRC Press, 2018. 1500 p. DOI:10.1201/9781315119144
12. Jöreskog K.G. Simultaneous factor analysis in several populations. Psychometrika, 1971. Vol. 36, pp. 409—426. DOI:10.1007/BF02291366
13. Li H., Hunter C.V., Bialo J.A. A Revisit of Zumbo’s Third Generation DIF: How Are We Doing in Language Testing? Language Assessment Quarterly, 2022. Vol. 19, no. 1, pp. 27—53. DOI:10.1080/15434303.2021.1963253
14. Luong R., Flake J.K. Measurement invariance testing using confirmatory factor analysis and alignment optimization: A tutorial for transparent analysis planning and reporting. Psychological Methods, 2023. Vol. 28(4), pp. 905—924. DOI:10.1037/met0000441
15. Visser L., Cartschau F., Von Goldammer A., Brandenburg J., Timmerman M., Hasselhorn M., Mähler C. Measurement Invariance in Relation to First Language: An Evaluation of German Reading and Spelling Tests. Applied Measurement in Education, 2023. Vol. 36, no. 2, pp. 115—131. DOI:10.1080/08957347.2023.2201701
16. Leitgöb H., Seddig D., Asparouhov T. [et al.]. Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives. Social Science Research, 2023. Vol. 110, article ID 102805. 30 p. DOI:10.1016/j.ssresearch.2022.102805
17. Schnoor B., Hartig J., Klinger T., Naumann A., Usanova I. Measuring the development of general language skills in English as a foreign language—Longitudinal invariance of the C-test. Language Testing, 2023. Vol. 40, no. 3, pp. 796—819. DOI:10.1177/02655322231159829
18. Meredith W. Notes on factorial invariance. Psychometrika, 1964. Vol. 29, pp. 177—185. DOI:10.1007/BF02289699
19. Papadopoulos T.C., Csépe V., Aro M., Caravolas M., Diakidoy I.-A., Olive T. Methodological Issues in Literacy Research Across Languages: Evidence From Alphabetic Orthographies. Reading Research Quarterly, 2021. Vol. 56, no. S1, pp. S351—S370. DOI:10.1002/rrq.407
20. Miller L.A., Lovler R.L. Foundations of psychological testing: A practical approach. Thousand Oaks: SAGE Publications Inc, 2020. 592 p.
21. Millsap R.E., Meredith W. Factorial invariance: Historical perspectives and new problems. In Cudeck R., MacCallum R.C. (eds.), Factor analysis at 100: Historical developments and future directions. New York: Routledge, 2007, pp. 131—152. DOI:10.4324/9780203936764
22. Millsap R.E. Statistical approaches to measurement invariance. New York: Routledge, 2011. 368 p. DOI:10.4324/9780203821961
23. Molenaar D. A flexible moderated factor analysis approach to test for measurement invariance across a continuous variable. Psychological Methods, 2020. Vol. 26(6), pp. 660—679. DOI:10.1037/met0000360
24. Welzel C., Brunkert L., Kruse S., Inglehart R.F. Non-invariance? An Overstated Problem With Misconceived Causes. Sociological Methods & Research, 2023. Vol. 52, no. 3, pp. 1368—1400. DOI:10.1177/0049124121995521
25. Pedraza O., van de Vijver F.J.R. Bias, equivalence, and fairness. In Pedraza O. (ed.), Clinical cultural neuroscience: An integrative approach to cross-cultural neuropsychology. Oxford: Oxford University Press, 2020, pp. 252—275.
26. Putnick D.L., Bornstein M.H. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 2016. Vol. 41, pp. 71—90. DOI:10.1016/j.dr.2016.06.004
27. Robitzsch A., Lüdtke O. Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons. Structural Equation Modeling: A Multidisciplinary Journal, 2023. Vol. 30, no. 6, pp. 859—870. DOI:10.1080/10705511.2023.2191292
28. Svetina D., Rutkowski L., Rutkowski D. Multiple-Group Invariance with Categorical Outcomes Using Updated Guidelines: An Illustration Using Mplus and the lavaan/semTools Packages. Structural Equation Modeling: A Multidisciplinary Journal, 2020. Vol. 27, no. 1, pp. 111—130. DOI:10.1080/10705511.2019.1602776
29. Tay L., Meade A.W., Cao M. An Overview and Practical Guide to IRT Measurement Equivalence Analysis. Organizational Research Methods, 2015. Vol. 18, no. 1, pp. 3—46. DOI:10.1177/1094428114553062
30. The standards for educational and psychological testing [Electronic resource]. American Psychological Association. Washington, 2009. URL: https://www.apa.org/science/programs/testing/standards (Accessed 25.06.2024).
31. Maassen E., D’Urso E.D., Van Assen M.A.L.M., Nuijten M.B., De Roover K., Wicherts J.M. The dire disregard of measurement invariance testing in psychological science. Psychological Methods, 2023. 14 p. DOI:10.1037/met0000624
32. Van de Vijver F.J., Leung K. Methods and data analysis for cross-cultural research. Cambridge: Cambridge University Press, 2021. 250 p. DOI:10.1017/9781107415188
33. Vandenberg R.J., Lance C.E. A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research. Organizational Research Methods, 2000. Vol. 3, no. 1, pp. 4—70. DOI:10.1177/109442810031002
34. Meuleman B., Żółtak T., Pokropek A., Davidov E., Muthén B., Oberski D.L., Billiet J., Schmidt P. Why Measurement Invariance is Important in Comparative Research. A Response to Welzel et al (2021). Sociological Methods & Research, 2023. Vol. 52, no. 3, pp. 1401—1419. DOI:10.1177/00491241221091755
35. Wicherts J.M. The importance of measurement invariance in neurocognitive ability testing. The Clinical Neuropsychologist, 2016. Vol. 30, no. 7, pp. 1006—1016. DOI:10.1080/13854046.2016.1205136
36. Widaman K.F., Ferrer E., Conger R.D. Factorial Invariance Within Longitudinal Structural Equation Models: Measuring the Same Construct Across Time. Child Development Perspectives, 2010. Vol. 4, no. 1, pp. 10—18. DOI:10.1111/j.1750-8606.2009.00110.x
37. Widaman K.F., Reise S.P. Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In Bryant K.J., Windle M., West S.G. (eds.), The science of prevention: Methodological advances from alcohol and substance abuse research. Washington: American Psychological Association, 1997, pp. 281—324. DOI:10.1037/10222-009
38. Wingen T., Berkessel J.B., Englich B. No Replication, No Trust? How Low Replicability Influences Trust in Psychology. Social Psychological and Personality Science, 2020. Vol. 11, no. 4, pp. 454—463. DOI:10.1177/1948550619877412