Journal of Modern Foreign Psychology
2024. Vol. 13, no. 2, 62–70
doi:10.17759/jmfp.2024130206
ISSN: 2304-4977 (online)
Testing the Tests: Investigating Measurement Invariance in Assessments of Cognitive and Language Abilities
Abstract
This paper discusses the measurement invariance of assessment instruments. The purpose of measurement invariance testing is to assess the extent to which a measured construct has comparable meaning across different groups or measurement points, to ensure that comparisons of these groups or measures to each other are appropriate. The paper provides an overview of recent and classic papers discussing conceptual and methodological aspects of invariance testing. Specific focus is placed on the available data on invariance testing for the assessments of cognitive and language abilities, and on the significance of invariance for the accurate assessment of individual and between-group differences. Finally, current approaches to invariance testing within structural equation modelling and item response theory are discussed. This paper may serve as an introduction to the concept of measurement invariance. It will be useful to readers interested in improving the quality of measurement instruments and making more reliable inferences about psychological phenomena.
General Information
Keywords: measurement invariance, psychometrics, psychological assessment, between-group differences, differential test functioning
Journal rubric: General Psychology
Article type: review article
DOI: https://doi.org/10.17759/jmfp.2024130206
Funding. This work is supported by the Russian Science Foundation (project № 23-28-01399).
Acknowledgements. The author is grateful for supervision to her doctoral advisor, Dr. E.L. Grigorenko.
Received: 05.05.2024
Accepted:
For citation: Logvinenko T.I. Testing the Tests: Investigating Measurement Invariance in Assessments of Cognitive and Language Abilities [Elektronnyi resurs]. Sovremennaia zarubezhnaia psikhologiia = Journal of Modern Foreign Psychology, 2024. Vol. 13, no. 2, pp. 62–70. DOI: 10.17759/jmfp.2024130206.
References
- Schneider L., Strobl C., Zeileis A., Debelak R. An R toolbox for score-based measurement invariance tests in IRT models. Behavior Research Methods, 2021. Vol. 54, pp. 2101—2113. DOI:10.3758/s13428-021-01689-0
- Bauer D.J., Belzak W.C.M., Cole V.T. Simplifying the Assessment of Measurement Invariance over Multiple Background Variables: Using Regularized Moderated Nonlinear Factor Analysis to Detect Differential Item Functioning. Structural Equation Modeling: A Multidisciplinary Journal, 2020. Vol. 27, no. 1, pp. 43—55. DOI:10.1080/10705511.2019.1642754
- Boer D., Hanke K., He J. On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests. Journal of Cross-Cultural Psychology, 2018. Vol. 49, no. 5, pp. 713—734. DOI:10.1177/0022022117749042
- Choi Y.-J., Asilkalkan A.R. Packages for Item Response Theory Analysis: Descriptions and Features. Measurement: Interdisciplinary Research and Perspectives, 2019. Vol. 17, no. 3, pp. 168—175. DOI:10.1080/15366367.2019.1586404
- Talbott E., De Los Reyes A., Kearns D.M., Mancilla-Martinez J., Wang M. Evidence-Based Assessment in Special Education Research: Advancing the Use of Evidence in Assessment Tools and Empirical Processes. Exceptional Children, 2023. Vol. 89, no. 4, pp. 467—487. DOI:10.1177/00144029231171092
- Jonson J.L., Geisinger K.F. (eds.), Fairness in educational and psychological testing: examining theoretical, research, practice, and policy implications of the 2014 standards. Washington: American Educational Research Association, 2022. 466 p. DOI:10.2307/j.ctv2kzv0fw
- Flake J.K., Fried E.I. Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science, 2020. Vol. 3, no. 4, pp. 456—465. DOI:10.1177/2515245920952393
- Funder D.C., Gardiner G. MIsgivings about measurement invariance. European Journal of Personality, 2024. 7 p. Ahead of Print. DOI:10.1177/08902070241228338
- Hajovsky D.B., Chesnut S.R. Examination of differential effects of cognitive abilities on reading and mathematics achievement across race and ethnicity: Evidence with the WJ IV. Journal of School Psychology, 2022. Vol. 93, pp. 1—27. DOI:10.1016/j.jsp.2022.05.001
- Han K., Colarelli S.M., Weed N.C. Methodological and statistical advances in the consideration of cultural diversity in assessment: A critical review of group classification and measurement invariance testing. Psychological Assessment, 2019. Vol. 31(12), pp. 1481—1496. DOI:10.1037/pas0000731
- Van Der Linden W.J. (ed.), Handbook of Item Response Theory: Three Volume Set. Boca Raton: CRC Press, 2018. 1500 p. DOI:10.1201/9781315119144
- Jöreskog K.G. Simultaneous factor analysis in several populations. Psychometrika, 1971. Vol. 36, pp. 409—426. DOI:10.1007/BF02291366
- Li H., Hunter C.V., Bialo J.A. A Revisit of Zumbo’s Third Generation DIF: How Are We Doing in Language Testing? Language Assessment Quarterly, 2022. Vol. 19, no. 1, pp. 27—53. DOI:10.1080/15434303.2021.1963253
- Luong R., Flake J.K. Measurement invariance testing using confirmatory factor analysis and alignment optimization: A tutorial for transparent analysis planning and reporting. Psychological Methods, 2023. Vol. 28(4), pp. 905—924. DOI:10.1037/met0000441
- Visser L., Cartschau F., Von Goldammer A., Brandenburg J., Timmerman M., Hasselhorn M., Mähler C. Measurement Invariance in Relation to First Language: An Evaluation of German Reading and Spelling Tests. Applied Measurement in Education, 2023. Vol. 36, no. 2, pp. 115—131. DOI:10.1080/08957347.2023.2201701
- Leitgöb H., Seddig D., Asparouhov T. [et al.]. Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives. Social Science Research, 2023. Vol. 110, article ID 102805. 30 p. DOI:10.1016/j.ssresearch.2022.102805
- Schnoor B., Hartig J., Klinger T., Naumann A., Usanova I. Measuring the development of general language skills in English as a foreign language—Longitudinal invariance of the C-test. Language Testing, 2023. Vol. 40, no. 3, pp. 796—819. DOI:10.1177/02655322231159829
- Meredith W. Notes on factorial invariance. Psychometrika, 1964. Vol. 29, pp. 177—185. DOI:10.1007/BF02289699
- Papadopoulos T.C., Csépe V., Aro M., Caravolas M., Diakidoy I.-A., Olive T. Methodological Issues in Literacy Research Across Languages: Evidence From Alphabetic Orthographies. Reading Research Quarterly, 2021. Vol. 56, no. S1, pp. S351—S370. DOI:10.1002/rrq.407
- Miller L.A., Lovler R.L. Foundations of psychological testing: A practical approach. Thousand Oaks: SAGE Publications Inc, 2020. 592 p.
- Millsap R.E., Meredith W. Factorial invariance: Historical perspectives and new problems. In Cudeck R., MacCallum R.C. (eds.), Factor analysis at 100: Historical developments and future directions. New York: Routledge, 2007, pp. 131—152. DOI:10.4324/9780203936764
- Millsap R.E. Statistical approaches to measurement invariance. New York: Routledge, 2011. 368 p. DOI:10.4324/9780203821961
- Molenaar D. A flexible moderated factor analysis approach to test for measurement invariance across a continuous variable. Psychological Methods, 2020. Vol. 26(6), pp. 660—679. DOI:10.1037/met0000360
- Welzel C., Brunkert L., Kruse S., Inglehart R.F. Non-invariance? An Overstated Problem With Misconceived Causes. Sociological Methods & Research, 2023. Vol. 52, no. 3, pp. 1368—1400. DOI:10.1177/0049124121995521
- Pedraza O., van de Vijver F.J.R. Bias, equivalence, and fairness. In Pedraza O. (ed.), Clinical cultural neuroscience: An integrative approach to cross-cultural neuropsychology. Oxford: Oxford University Press, 2020, pp. 252—275.
- Putnick D.L., Bornstein M.H. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 2016. Vol. 41, pp. 71—90. DOI:10.1016/j.dr.2016.06.004
- Robitzsch A., Lüdtke O. Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons. Structural Equation Modeling: A Multidisciplinary Journal, 2023. Vol. 30, no. 6, pp. 859—870. DOI:10.1080/10705511.2023.2191292
- Svetina D., Rutkowski L., Rutkowski D. Multiple-Group Invariance with Categorical Outcomes Using Updated Guidelines: An Illustration Using Mplus and the lavaan/semTools Packages. Structural Equation Modeling: A Multidisciplinary Journal, 2020. Vol. 27, no. 1, pp. 111—130. DOI:10.1080/10705511.2019.1602776
- Tay L., Meade A.W., Cao M. An Overview and Practical Guide to IRT Measurement Equivalence Analysis. Organizational Research Methods, 2015. Vol. 18, no. 1, pp. 3—46. DOI:10.1177/1094428114553062
- The standards for educational and psychological testing [Electronic resource]. American Psychological Association. Washington, 2009. URL: https://www.apa.org/science/programs/testing/standards (Accessed 25.06.2024).
- Maassen E., D’Urso E.D., Van Assen M.A.L.M., Nuijten M.B., De Roover K., Wicherts J.M. The dire disregard of measurement invariance testing in psychological science. Psychological Methods, 2023. 14 p. DOI:10.1037/met0000624
- Van de Vijver F.J., Leung K. Methods and data analysis for cross-cultural research. Cambridge: Cambridge University Press, 2021. 250 p. DOI:10.1017/9781107415188
- Vandenberg R.J., Lance C.E. A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research. Organizational Research Methods, 2000. Vol. 3, no. 1, pp. 4—70. DOI:10.1177/109442810031002
- Meuleman B., Żółtak T., Pokropek A., Davidov E., Muthén B., Oberski D.L., Billiet J., Schmidt P. Why Measurement Invariance is Important in Comparative Research. A Response to Welzel et al (2021). Sociological Methods & Research, 2023. Vol. 52, no. 3, pp. 1401—1419. DOI:10.1177/00491241221091755
- Wicherts J.M. The importance of measurement invariance in neurocognitive ability testing. The Clinical Neuropsychologist, 2016. Vol. 30, no. 7, pp. 1006—1016. DOI:10.1080/13854046.2016.1205136
- Widaman K.F., Ferrer E., Conger R.D. Factorial Invariance Within Longitudinal Structural Equation Models: Measuring the Same Construct Across Time. Child Development Perspectives, 2010. Vol. 4, no. 1, pp. 10—18. DOI:10.1111/j.1750-8606.2009.00110.x
- Widaman K.F., Reise S.P. Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In Bryant K.J., Windle M., West S.G. (eds.), The science of prevention: Methodological advances from alcohol and substance abuse research. Washington: American Psychological Association, 1997, pp. 281—324. DOI:10.1037/10222-009
- Wingen T., Berkessel J.B., Englich B. No Replication, No Trust? How Low Replicability Influences Trust in Psychology. Social Psychological and Personality Science, 2020. Vol. 11, no. 4, pp. 454—463. DOI:10.1177/1948550619877412
Information About the Authors
Metrics
Views
Total: 156
Previous month: 24
Current month: 14
Downloads
Total: 53
Previous month: 3
Current month: 6