Testing the Tests: Investigating Measurement Invariance in Assessments of Cognitive and Language Abilities

53

Abstract

This paper discusses the measurement invariance of assessment instruments. The purpose of measurement invariance testing is to assess the extent to which a measured construct has comparable meaning across different groups or measurement points, to ensure that comparisons of these groups or measures to each other are appropriate. The paper provides an overview of recent and classic papers discussing conceptual and methodological aspects of invariance testing. Specific focus is placed on the available data on invariance testing for the assessments of cognitive and language abilities, and on the significance of invariance for the accurate assessment of individual and between-group differences. Finally, current approaches to invariance testing within structural equation modelling and item response theory are discussed. This paper may serve as an introduction to the concept of measurement invariance. It will be useful to readers interested in improving the quality of measurement instruments and making more reliable inferences about psychological phenomena.

General Information

Keywords: measurement invariance, psychometrics, psychological assessment, between-group differences, differential test functioning

Journal rubric: General Psychology

Article type: review article

DOI: https://doi.org/10.17759/jmfp.2024130206

Funding. This work is supported by the Russian Science Foundation (project № 23-28-01399).

Acknowledgements. The author is grateful for supervision to her doctoral advisor, Dr. E.L. Grigorenko.

Received: 05.05.2024

Accepted:

For citation: Logvinenko T.I. Testing the Tests: Investigating Measurement Invariance in Assessments of Cognitive and Language Abilities [Elektronnyi resurs]. Sovremennaia zarubezhnaia psikhologiia = Journal of Modern Foreign Psychology, 2024. Vol. 13, no. 2, pp. 62–70. DOI: 10.17759/jmfp.2024130206.

References

  1. Schneider L., Strobl C., Zeileis A., Debelak R. An R toolbox for score-based measurement invariance tests in IRT models. Behavior Research Methods, 2021. Vol. 54, pp. 2101—2113. DOI:10.3758/s13428-021-01689-0
  2. Bauer D.J., Belzak W.C.M., Cole V.T. Simplifying the Assessment of Measurement Invariance over Multiple Background Variables: Using Regularized Moderated Nonlinear Factor Analysis to Detect Differential Item Functioning. Structural Equation Modeling: A Multidisciplinary Journal, 2020. Vol. 27, no. 1, pp. 43—55. DOI:10.1080/10705511.2019.1642754
  3. Boer D., Hanke K., He J. On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests. Journal of Cross-Cultural Psychology, 2018. Vol. 49, no. 5, pp. 713—734. DOI:10.1177/0022022117749042
  4. Choi Y.-J., Asilkalkan A.R. Packages for Item Response Theory Analysis: Descriptions and Features. Measurement: Interdisciplinary Research and Perspectives, 2019. Vol. 17, no. 3, pp. 168—175. DOI:10.1080/15366367.2019.1586404
  5. Talbott E., De Los Reyes A., Kearns D.M., Mancilla-Martinez J., Wang M. Evidence-Based Assessment in Special Education Research: Advancing the Use of Evidence in Assessment Tools and Empirical Processes. Exceptional Children, 2023. Vol. 89, no. 4, pp. 467—487. DOI:10.1177/00144029231171092
  6. Jonson J.L., Geisinger K.F. (eds.), Fairness in educational and psychological testing: examining theoretical, research, practice, and policy implications of the 2014 standards. Washington: American Educational Research Association, 2022. 466 p. DOI:10.2307/j.ctv2kzv0fw
  7. Flake J.K., Fried E.I. Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science, 2020. Vol. 3, no. 4, pp. 456—465. DOI:10.1177/2515245920952393
  8. Funder D.C., Gardiner G. MIsgivings about measurement invariance. European Journal of Personality, 2024. 7 p. Ahead of Print. DOI:10.1177/08902070241228338
  9. Hajovsky D.B., Chesnut S.R. Examination of differential effects of cognitive abilities on reading and mathematics achievement across race and ethnicity: Evidence with the WJ IV. Journal of School Psychology, 2022. Vol. 93, pp. 1—27. DOI:10.1016/j.jsp.2022.05.001
  10. Han K., Colarelli S.M., Weed N.C. Methodological and statistical advances in the consideration of cultural diversity in assessment: A critical review of group classification and measurement invariance testing. Psychological Assessment, 2019. Vol. 31(12), pp. 1481—1496. DOI:10.1037/pas0000731
  11. Van Der Linden W.J. (ed.), Handbook of Item Response Theory: Three Volume Set. Boca Raton: CRC Press, 2018. 1500 p. DOI:10.1201/9781315119144
  12. Jöreskog K.G. Simultaneous factor analysis in several populations. Psychometrika, 1971. Vol. 36, pp. 409—426. DOI:10.1007/BF02291366
  13. Li H., Hunter C.V., Bialo J.A. A Revisit of Zumbo’s Third Generation DIF: How Are We Doing in Language Testing? Language Assessment Quarterly, 2022. Vol. 19, no. 1, pp. 27—53. DOI:10.1080/15434303.2021.1963253
  14. Luong R., Flake J.K. Measurement invariance testing using confirmatory factor analysis and alignment optimization: A tutorial for transparent analysis planning and reporting. Psychological Methods, 2023. Vol. 28(4), pp. 905—924. DOI:10.1037/met0000441
  15. Visser L., Cartschau F., Von Goldammer A., Brandenburg J., Timmerman M., Hasselhorn M., Mähler C. Measurement Invariance in Relation to First Language: An Evaluation of German Reading and Spelling Tests. Applied Measurement in Education, 2023. Vol. 36, no. 2, pp. 115—131. DOI:10.1080/08957347.2023.2201701
  16. Leitgöb H., Seddig D., Asparouhov T. [et al.]. Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives. Social Science Research, 2023. Vol. 110, article ID 102805. 30 p. DOI:10.1016/j.ssresearch.2022.102805
  17. Schnoor B., Hartig J., Klinger T., Naumann A., Usanova I. Measuring the development of general language skills in English as a foreign language—Longitudinal invariance of the C-test. Language Testing, 2023. Vol. 40, no. 3, pp. 796—819. DOI:10.1177/02655322231159829
  18. Meredith W. Notes on factorial invariance. Psychometrika, 1964. Vol. 29, pp. 177—185. DOI:10.1007/BF02289699
  19. Papadopoulos T.C., Csépe V., Aro M., Caravolas M., Diakidoy I.-A., Olive T. Methodological Issues in Literacy Research Across Languages: Evidence From Alphabetic Orthographies. Reading Research Quarterly, 2021. Vol. 56, no. S1, pp. S351—S370. DOI:10.1002/rrq.407
  20. Miller L.A., Lovler R.L. Foundations of psychological testing: A practical approach. Thousand Oaks: SAGE Publications Inc, 2020. 592 p.
  21. Millsap R.E., Meredith W. Factorial invariance: Historical perspectives and new problems. In Cudeck R., MacCallum R.C. (eds.), Factor analysis at 100: Historical developments and future directions. New York: Routledge, 2007, pp. 131—152. DOI:10.4324/9780203936764
  22. Millsap R.E. Statistical approaches to measurement invariance. New York: Routledge, 2011. 368 p. DOI:10.4324/9780203821961
  23. Molenaar D. A flexible moderated factor analysis approach to test for measurement invariance across a continuous variable. Psychological Methods, 2020. Vol. 26(6), pp. 660—679. DOI:10.1037/met0000360
  24. Welzel C., Brunkert L., Kruse S., Inglehart R.F. Non-invariance? An Overstated Problem With Misconceived Causes. Sociological Methods & Research, 2023. Vol. 52, no. 3, pp. 1368—1400. DOI:10.1177/0049124121995521
  25. Pedraza O., van de Vijver F.J.R. Bias, equivalence, and fairness. In Pedraza O. (ed.), Clinical cultural neuroscience: An integrative approach to cross-cultural neuropsychology. Oxford: Oxford University Press, 2020, pp. 252—275.
  26. Putnick D.L., Bornstein M.H. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 2016. Vol. 41, pp. 71—90. DOI:10.1016/j.dr.2016.06.004
  27. Robitzsch A., Lüdtke O. Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons. Structural Equation Modeling: A Multidisciplinary Journal, 2023. Vol. 30, no. 6, pp. 859—870. DOI:10.1080/10705511.2023.2191292
  28. Svetina D., Rutkowski L., Rutkowski D. Multiple-Group Invariance with Categorical Outcomes Using Updated Guidelines: An Illustration Using Mplus and the lavaan/semTools Packages. Structural Equation Modeling: A Multidisciplinary Journal, 2020. Vol. 27, no. 1, pp. 111—130. DOI:10.1080/10705511.2019.1602776
  29. Tay L., Meade A.W., Cao M. An Overview and Practical Guide to IRT Measurement Equivalence Analysis. Organizational Research Methods, 2015. Vol. 18, no. 1, pp. 3—46. DOI:10.1177/1094428114553062
  30. The standards for educational and psychological testing [Electronic resource]. American Psychological Association. Washington, 2009. URL: https://www.apa.org/science/programs/testing/standards (Accessed 25.06.2024).
  31. Maassen E., D’Urso E.D., Van Assen M.A.L.M., Nuijten M.B., De Roover K., Wicherts J.M. The dire disregard of measurement invariance testing in psychological science. Psychological Methods, 2023. 14 p. DOI:10.1037/met0000624
  32. Van de Vijver F.J., Leung K. Methods and data analysis for cross-cultural research. Cambridge: Cambridge University Press, 2021. 250 p. DOI:10.1017/9781107415188
  33. Vandenberg R.J., Lance C.E. A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research. Organizational Research Methods, 2000. Vol. 3, no. 1, pp. 4—70. DOI:10.1177/109442810031002
  34. Meuleman B., Żółtak T., Pokropek A., Davidov E., Muthén B., Oberski D.L., Billiet J., Schmidt P. Why Measurement Invariance is Important in Comparative Research. A Response to Welzel et al (2021). Sociological Methods & Research, 2023. Vol. 52, no. 3, pp. 1401—1419. DOI:10.1177/00491241221091755
  35. Wicherts J.M. The importance of measurement invariance in neurocognitive ability testing. The Clinical Neuropsychologist, 2016. Vol. 30, no. 7, pp. 1006—1016. DOI:10.1080/13854046.2016.1205136
  36. Widaman K.F., Ferrer E., Conger R.D. Factorial Invariance Within Longitudinal Structural Equation Models: Measuring the Same Construct Across Time. Child Development Perspectives, 2010. Vol. 4, no. 1, pp. 10—18. DOI:10.1111/j.1750-8606.2009.00110.x
  37. Widaman K.F., Reise S.P. Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In Bryant K.J., Windle M., West S.G. (eds.), The science of prevention: Methodological advances from alcohol and substance abuse research. Washington: American Psychological Association, 1997, pp. 281—324. DOI:10.1037/10222-009
  38. Wingen T., Berkessel J.B., Englich B. No Replication, No Trust? How Low Replicability Influences Trust in Psychology. Social Psychological and Personality Science, 2020. Vol. 11, no. 4, pp. 454—463. DOI:10.1177/1948550619877412

Information About the Authors

Tatiana I. Logvinenko, Clinical Psychologist, Research Fellow, University Hospital, Ludwig Maximilian University of Munich (LMU), National Research University Higher School of Economics, Munich, Germany, ORCID: https://orcid.org/0000-0001-7430-1963, e-mail: logvinenkota.spb@gmail.com

Metrics

Views

Total: 156
Previous month: 24
Current month: 14

Downloads

Total: 53
Previous month: 3
Current month: 6