Critical Thinking Assessment in Adults: Methodology and Development Experience

Critical thinking (next CT) is one of the key competencies in the modern world. Despite the variety of assessment systems for secondary and higher education, there is a lack of tests for assessing this competence for adults. The importance of adult assessment is emphasized primarily by the need for employers to have employees with developed CT in various work contexts.

The article describes the procedure for creating a measurement test for students of the training course «Critical Thinking» of the «Yandex.Practice» platform and the results of assessing its psychometric quality. When creating this test, we relied on the Evidence Centered Design (ECD) approach. The test is performed in computer form and consists of tasks aimed at identifying the relevant observed behavior of the subject. The validity of the developed test, including its compliance with theoretical expectations about the factor structure, is confirmed by evidence of constructive, criterion, convergent and content validity. The limitations of this study and possible directions for future developments are considered. The result of the work is a validated test for measuring critical thinking for adults, the quality of the tasks of which is confirmed by psychometric analysis in modern mental test theory (IRT — Item Response Theory).

1. Volkov E.N. Testy kriticheskogo myshleniya: vvodnyi obzor [Critical thinking tests] [Elektronnyi resurs]. Psikhologicheskaya diagnostika [Psychological diagnostics], 2015. Vol. 3, pp. 5—23. URL: https://evolkov.net/critic.think/tests/Critic.Think.Tests.Volkov.E.2015.pdf (Accessed 01.11.2022). (In Russ.).
2. Koreshnikova Yu.N., Froumin I.D. Professional'nye kompetentsii pedagoga kak faktor sformirovannosti kriticheskogo myshleniya studentov [Teachers’ Professional Skills as a Factor in the Development of Students’ Critical Thinking]. Psikhologicheskaya nauka i obrazovanie = Psychological Science and Education, 2020. Vol. 25, no. 6, pp. 88—103. DOI:10.17759/pse.2020250608 (In Russ.).
3. Koreshnikova Yu.N., Froumin I.D., Pashchenko T.V. Organizatsionnye i pedagogicheskie usloviya formirovaniya navyka kriticheskogo myshleniya u studentov rossiiskikh vuzov [Organizational and pedagogical conditions for the development of critical thinking skills among russian university students]. Universitetskoe upravlenie: praktika i analiz = University Management: Practice and Analysis, 2021. Vol. 25, no. 1, pp. 5—17. DOI:10.15826/umpa.2021.01.001 (In Russ.).
4. Portal Federal'nykh gosudarstvennykh obrazovatel'nykh standartov vysshego obrazovaniya [Elektronnyi resurs] [Portal of the Federal State Educational Standards of Higher Education]. NITU «MISiS», 2022. URL: https://fgosvo.ru/fgosvo/index/24 (Accessed 01.11.2022). (In Russ.).
5. Uglanova I.L., Orel E.A., Brun I.V. Izmerenie kreativnosti i kriticheskogo myshleniya v nachal'noi shkole [Measuring creativityand critical thinking in primary school]. Psikhologicheskii zhurnal = Psychological journal, 2020. Vol. 41, no. 6, pp. 96—107. DOI:10.31857/S020595920011124-2 (In Russ.).
6. Dobryakova M.S., Froumin I.D., Barannikov K.A., Remorenko I.M., Ziil N., Moss D., Khautamyaki Ya. Universal'nye kompetentnosti i novaya gramotnost': ot lozungov k real'nosti [Universal competencies and new literacy: from slogans to reality]. Moscow: Izd. dom Vysshei shkoly ekonomiki, 2020. 468 p. DOI:10.17323/978-5-7598-2177-9 (In Russ.).
7. Ab Kadir M.A. What Teacher Knowledge Matters in Effectively Developing Critical Thinkers in the 21st Century Curriculum? Thinking Skills and Creativity, 2017. Vol. 23, pp. 79—90. DOI:10.1016/j.tsc.2016.10.011
8. Akoglu H. User's guide to correlation coefficients. Turkish journal of emergency medicine, 2018. Vol. 18(3), pp. 91—93. DOI:10.1016/j.tjem.2018.08.001
9. Shavelson R.J., Zlatkin-Troitschanskaia O., Beck K., Schmidt S., Marino J.P. Assessment of critical thinking: Next generation performance assessment. International Journal of Testing, 2019. Vol. 19(9), pp. 337—362. DOI:10.1080/15305058.2018.1543309
10. Bandyopadhyay S., Szostek J. Thinking critically about critical thinking: Assessing critical thinking of business students using multiple measures. Journal of Education for Business, 2019. Vol. 94(4), pp. 259—270. DOI:10.1080/08832323.2018.1524355
11. Bruine de Bruin W., Parker A.M., Fischhoff B. Individual differences in adult decision-making competence. Journal of personality and social psychology, 2007. Vol. 92, no. 5, pp. 938—956. DOI:10.1037/0022-3514.92.5.938
12. Butler H.A. Halpern Critical Thinking Assessment predicts real world outcomes of critical thinking. Applied Cognitive Psychology, 2012. Vol. 26, no. 5, pp. 721—729. DOI:10.1002/acp.2851
13. Calma A., Cotronei-Baird V. Assessing critical thinking in business education: Key issues and practical solutions. The International Journal of Management Education, 2021. Vol. 19, no. 3, article ID 100531. 14 p. DOI:10.1016/j.ijme.2021.100531
14. Sobocan J. (eds.) Critical thinking education and assessment: 2nd ed. Windsor: Windsor Studies in Argumentation, 2022. 324 p. DOI:10.22329/wsia.12.2022
15. Dwyer C.P., Walsh A. An exploratory quantitative case study of critical thinking development through adult distance learning. Educational Technology Research and Development, 2020. Vol. 68(1), pp. 17—35. DOI:10.1007/s11423-019-09659-2
16. Plummer K.J., Kebritchi M., Leary H.M., Halverson D.M. Enhancing Critical Thinking Skills through Decision-Based Learning. Innovative Higher Education, 2022. Vol. 47(2), pp. 711—734. DOI:10.1007/s10755-022-09595-9
17. Hitchcock D. “Critical Thinking” [Elektronnyi resurs]. In Edward N. Zalta, Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy (Fall 2020 Edition). Stanford: The Metaphysics Research Lab, 2022. URL: https://plato.stanford.edu/entries/critical-thinking/ (Accessed 24.06.2022).
18. Chen W.H., Lenderking W., Jin Y., Wyrwich K.W., Gelhorn H., Revicki D.A. Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Quality of Life Research, 2014. Vol. 23(2), pp. 485—493. DOI:10.1007/s11136-013-0487-5
19. Kay C.S., Slovic P. The Generic Conspiracist Beliefs Scale—5: A short-form measure of conspiracist ideation. PsyArXiv, 2022. 72 p. DOI:10.31234/osf.io/2uvr9
20. Leighton J.P. Using think-aloud interviews and cognitive labs in educational research. Oxford: Oxford University Press, 2017. 216 p. DOI:10.1093/acprof:oso/9780199372904.001.0001
21. Linacre J.M. Optimizing rating scale category effectiveness [Elektronnyi resurs]. Journal of applied measurement, 2002. Vol. 3(1), pp. 85—106. URL: https://psycnet.apa.org/record/2002-01669-005 (Accessed 24.06.2022).
22. Masters G.N. A Rasch model for partial credit scoring. Psychometrika, 1982. Vol. 47(2), pp. 149—174. DOI:10.1007/BF02296272
23. Mehmet Ş.A.H.I.N. Critical thinking and transformative learning [Elektronnyi resurs]. Journal of Innovation in Psychology, Education and Didactics, 2018. Vol. 22(1), pp. 103—114. URL: https://files.eric.ed.gov/fulltext/ED593584.pdf (Accessed 08.11.22).
24. Mislevy R.J. Sociocognitive Foundations of Educational Measurement. N.Y.: Routledge, 2018. 458 p. DOI:10.4324/9781315871691
25. Mislevy R.J., Almond R.G., Lukas J.F. A brief introduction to evidence centered design. ETS Research Report Series, 2014. Vol. 2003, no. 1, pp. i-29. DOI:10.1002/j.2333-8504.2003.tb01908.x
26. Oliveri M.E., Mislevy R.J. Introduction to “Challenges and Opportunities in the Design of ‘Next-Generation Assessments of 21st Century Skills’” Special Issue. International Journal of Testing, 2019. Vol. 19, no. 2, pp. 97—102. DOI:10.1080/15305058.2019.1608551
27. Penkauskienė D., Railienė A., Cruz G. How is critical thinking valued by the labour market? Employer perspectives from different European countries. Studies in Higher Education, 2019. Vol. 44(5), pp. 804—815. DOI:10.1080/03075079.2019.1586323
28. Smith Jr E.V. Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. Journal of applied measurement, 2002. Vol. 3(2), pp. 205—231.
29. Sujová E., Čierna H., Simanová Ľ., Gejdoš P., Štefková J. Soft Skills Integration into Business Processes Based on the Requirements of Employers-Approach for Sustainable Education. Sustainability, 2021. Vol. 13(24), article ID 13807. 13 p. DOI:10.3390/su132413807
30. American Educational Research Association, American Psychological Association & National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, D.C: American Educational Research Association, 2014. 230 p.
31. Blair J.A. Studies in Critical Thinking: 2nd Edition. Windsor, Ontario: Windsor Studies in Argumentation, 2019. 393 p. DOI:10.22329/wsia.08.2019
32. Van Laar E., Van Deursen A.J., Van Dijk J.A., De Haan J. The relation between 21st-century skills and digital skills: A systematic literature review. Computers in human behavior, 2017. Vol. 72, pp. 577—588. DOI:10.1016/j.chb.2017.03.010
33. Shaw A., Liu O. L., Gu L. et all. Thinking critically about critical thinking: validating the Russian HEIghten® critical thinking assessment. Studies in Higher Education, 2020. Vol. 45, no. 9, pp. 1933—1948. DOI:10.1080/03075079.2019.1672640
34. West R.F., Toplak M.E., Stanovich K.E. Heuristics and biases as measures of critical thinking: associations with cognitive ability and thinking dispositions. Journal of Educational Psychology, 2008. Vol. 100(4), pp. 930—941. DOI:10.1037/a0012842