Evidence-Centered Design method for measuring complex psychological constructs

1870

Abstract

The article describes validation process of the diagnostic tool for monitoring assessment of socio-emotional habits in elementary school. The tool is based on the Big Five model and includes three scales: Goal achievement, Cooperation and Emotional control. Two validation researches have been conducted, both on third-grade students from Moscow and Tatarstan elementary schools. Preliminary research (N=1318) examined the construct validity and reliability of the questionnaire. Confirmatory and exploratory factor analysis and methods of Modern Test Theory have confirmed hypothecated factorial structure. Subsequent research (N=2559) proved criterion validity of the questionnaire: it showed expected level and direction of correlation with the results of reading and mathematical skills testing. Presented questionnaire can be used for group monitoring assessment of socio-emotional development in elementary school.

General Information

Keywords: socio-emotional habits, elementary school, reliability, validity, psychometric analysis

Journal rubric: Labour Psychology and Engineering Psychology

Article type: scientific article

DOI: https://doi.org/10.17759/jmfp.2018070302

For citation: Uglanova I.L., Brun I.V., Vasin G.M. Evidence-Centered Design method for measuring complex psychological constructs [Elektronnyi resurs]. Sovremennaia zarubezhnaia psikhologiia = Journal of Modern Foreign Psychology, 2018. Vol. 7, no. 3, pp. 18–27. DOI: 10.17759/jmfp.2018070302. (In Russ., аbstr. in Engl.)

References

  1. Boyatsis R. Kompetentnyi menedzher. Model' effektivnoi raboty [Competent manager. Model of effective work]. Moscow: KhIPPO, 2008. 340 p. (In Russ.).
  2. Avdeeva S.M. et al. Otsenka informatsionno-kommunikatsionnoi kompetentnosti uchashchikhsya: podkhody, instrument, validnost' i nadezhnost' rezul'tatov [Assessment of information and communication competence of students: approaches, tools, validity and reliability of results] [Elektronnyi resurs]. Voprosy obrazovaniya [Education issues], 2017, no. 4, pp. 104–132. Available at: https://cyberleninka.ru/article/n/otsenka-informatsionno-kommunikatsionnoy-kompetentnosti-uchaschihsya-podhody-instrument-validnost-i-nadezhnost-rezultatov (Accessed 04.07.2018). (In Russ.; Abstr. in Engl.).
  3. Khlopotov M.V. Primenenie baiesovskoi seti pri postroenii modelei dlya otsenki urovnya sformirovannosti kompetentsii [Bayesian network application in building models for assessing the level of competence formation] [Elektronnyi resurs]. Internet-zhurnal Naukovedenie [Internet Journal of Science], 2014, no. 5(24), pp. 1–28. Available at: https://cyberleninka.ru/article/n/primenenie-bayesovskoy-seti-pri-postroenii-modeley-dlya-otsenki-urovnya-sformirovannosti-kompetentsiy (Accessed 26.08.2018). (In Russ.).
  4. Almond R., Steinberg L., Mislevy R. Enhancing the design and delivery of assessment systems: A four-process architecture. The Journal of Technology, Learning and Assessment, 2002, vol. 1, no. 5, pp. 1–64. Available at: https://ejournals.bc.edu/ojs/index.php/jtla/article/view/1671/1509 (Accessed 26.08.2018).
  5. Behrens J.T. et al. An evidence centered design for learning and assessment in the digital world [Elektronnyi resurs]: CRESST Report 778. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST), 2010. 48 p. Available at: https://files.eric.ed.gov/fulltext/ED520431.pdf (Accessed 26.08.2018).
  6. Almond R.G. et al. Bayesian networks in educational assessment. New York: Springer,  2015. 662 p. doi:10.1007/978-1-4939-2125-6
  7. de Klerk S., Veldkamp B.P., Eggen T.J.H.M., Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example. Computers & education, 2015, vol. 85, pp. 23–34. doi:10.1016/j.compedu.2014.12.020
  8. Almond R.G. et al. Debugging the evidence chain [Elektronnyi resurs]. In Almond R.G., Mengshoel O. (eds.) Proceedings of the 2013 uai application workshops: Big data meet complex models and models for spatial, temporal and network data (uai2013aw). Seattle, WA: Association For Uncertainty In Artificial Intelligence, 2013. pp. 1–10. Available at: http://ceur-ws.org/Vol-1024/paper-01.pdf (Accessed 26.08.2018).
  9. Ennis R.H. Critical thinking assessment. Theory into practice, 1993, vol. 32, no. 3, pp. 179–186. doi:10.1080/00405849309543594
  10. Facione P. Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction [Elektronnyi resurs]. Newark: California Academic Press, 1990. 112 p.. Available at: https://philarchive.org/archive/FACCTA (Accessed 26.08.2018).
  11. Griffin P., Care E., McGaw B. The changing role of education and schools // Assessment and teaching of 21st century skills. Dordrecht: Springer, 2012. pp. 1–15. doi:10.1007/978-94-007-2324-5_1
  12. Duckworth A.L. et al. Grit: perseverance and passion for long-term goals. Journal of personality and social psychology, 2007, vol. 92, no. 6, pp. 1087–1101. doi:10.1037/0022-3514.92.6.1087
  13. Steinberg L.S. et al. Introduction to the biomass project: An illustration of evidence-centered assessment design and delivery capability [Elektronnyi resurs]: CSE Report 609. Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing. Los Angeles, CA, 2003. 57 p. Available at: https://files.eric.ed.gov/fulltext/ED483027.pdf (Accessed 26.08.2018).
  14. Lai E.R. Critical thinking: A literature review [Elektronnyi resurs]: Research Report. Vol. 6.. Pearson, 2011. 49 p. Available at: http://images.pearsonassessments.com/images/tmrs/CriticalThinkingReviewFINAL.pdf (Accessed 26.08.2018).
  15. Lewis A., Smith D. Defining higher order thinking. Theory into practice, 1993, vol. 32, no. 3, pp. 131–137. doi:10.1080/00405849309543588
  16. Liu O.L., Frankel L., Roohr K.C. Assessing Critical Thinking in Higher Education: Current State and Directions for Next Generation Assessment. ETS Research Report Series, 2014, vol. 2014, no. 1, pp. 1–23. doi:10.1002/ets2.12009
  17. Shute V.J. et al. Melding the power of serious games and embedded assessment to monitor and foster learning. In Ritterfeld U., Cody M., Vorderer P. (Eds.). Serious games: Mechanisms and effects. New York; London: Routledge; Taylor and Francis, 2009, pp. 295–321.
  18. Messick S. Alternative modes of assessment, uniform standards of validity. Princeton, N.J.: Educational Testing Service, 1994. 25 p. (ETS Research Report Series. № 2). doi:10.1002/j.2333-8504.1994.tb01634.x
  19. Mislevy R.J., Almond R.G., Lukas J.F. A brief introduction to evidence-centered design: Research Report. Princeton, N.J.: Educational Testing Service, 2003. 37 p. (ETS Research Report Series. Vol. 2003. № 1.). doi:10.1002/j.2333-8504.2003.tb01908.x
  20. Mislevy R.J., Haertel G.D. Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 2006, vol. 25, no. 4, pp. 6–20. doi:10.1111/j.1745-3992.2006.00075.x
  21. Mislevy R.J. et al. Modeling conditional probabilities in complex educational assessments [Elektronnyi resurs]. Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies, University of California. Los Angeles, 2001. 56 p. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.322.4516&rep=rep1&type=pdf (Accessed 26.08.2018).
  22. Paul R. Critical thinking: What, why, and how. New directions for community colleges, 1992, vol. 1992, no. 77, pp. 3–24. doi:10.1002/cc.36819927703
  23. Polyak S.T., von Davier A.A., Peterschmidt K. Computational Psychometrics for the Measurement of Collaborative Problem Solving Skills. Frontiers in psychology, 2017, vol. 8, pp. 20–29. doi:10.3389/fpsyg.2017.02029
  24. Mislevy R.J. et al. Psychometric considerations in game-based assessment. GlassLab Report, 2014. 158 p.
  25. Shute V.J., Rahimi S. Review of computer-based assessment for learning in elementary and secondary education. Journal of Computer Assisted Learning, 2017, vol. 33, no. 1, pp. 1–19. doi:10.1111/jcal.12172
  26. Toulmin S.E. The uses of argument. Cambridge: Cambridge University Press, 1958. 262 p.

Information About the Authors

Irina L. Uglanova, Junior Research Fellow, Centre for Psychometrics and Measurements in Education, National Research University Higher School of Economics, Moscow, Russia, ORCID: https://orcid.org/0000-0001-9117-5997, e-mail: iuglanova@hse.ru

Irina V. Brun, Head of Laboratory of New Constructs Measurement and Tests Designing, National Research University «Higher School of Economics», Moscow, Russia, e-mail: ibrun@hse.ru

Georgy M. Vasin, Junior Researcher at Laboratory of New Constructs Measurement and Tests Designing, National Research University «Higher School of Economics», Moscow, Russia, e-mail: gvasin@hse.ru

Metrics

Views

Total: 2374
Previous month: 24
Current month: 6

Downloads

Total: 1870
Previous month: 14
Current month: 1