Factor Structure of the Russian Version of the «Metacognitive Awareness Inventory»

Metacognitive processes are important for the success in the wide range of educational activities of youth and young adults. However, the positive correlations between metacognition and academic achievements are not high enough, and the instruments used in these studies might be the reason. We explored the factor structure of the Russian version of the questionnaire “Metacognitive Awareness Inventory” developed by G. Schraw and R. Dennison and adapted by A.V. Karpov and I.M. Skityaeva into Russian. The participants of our study were 527 residents of St. Petersburg, which were studying at the university at the time. Among them there were 366 students getting their first diploma and 161students getting their second diploma (average age 23.8 ± 8.8). In this article the authors present the results of a confirmatory factor analysis of four models, which are the most frequently used in foreign and Russian literature: unidimensional model; two different two-factor models; eight-factor model. Evaluation of the model fit indices for the four models showed that none of them were a god fit. We reduced the number of items of the questionnaire and re-implemented the factor analysis of these four models. The values of indicators of a good model fit improved. In the short version of the questionnaire “Metacognitive Awareness Inventory” the authors discovered two scales – knowledge of cognition and regulation of cognition, which included 8 subscales: declarative knowledge, procedural knowledge, conditional knowledge, planning, information management strategies, comprehension monitoring, debugging strategies, evaluation.

Introduction

The latest studies in psychological and educational research are largely focused on metacognition in learning [7; 34]. J. Flavell was a pioneer in metacognition studies; in the 1980s his followers defined metacognition as a mental activity aimed at investigating the cognitive processes, and actively controlling and managing those processes in order to achieve particular goals [15; 17]. In broad terms, non-Russian metacognition studies are dedicated to two types of mental activity: knowledge of cognition (metacognitive knowledge; awareness of one’s own cognitive processes) and regulation of cognition (metacognitive regulation; monitoring and control of one’s cognitive processes), which are often studied within the framework of the common phenomenon of metacognitive awareness [25; 27; 33]. This concept is currently being developed by researchers from all over the world.

Within the framework of the structural-dialectical approach, T. E. Chernokova described the detailed structure of metacognition, including metacognitive knowledge (knowledge of general and individual patterns, objective conditions, and instruments of cognition) and metacognitive processes (control, regulation, and management of cognition) [9]. The author defined metacognition as a system of one's knowledge about cognitive activity in general and one’s own cognitive processes, as well as the processes that ensure self-regulation of one’s cognitive activity [10]. M. A. Kholodnaya proposed the concept of “mental experience”, which included involuntary and voluntary intellectual control, an open cognitive position, and metacognitive awareness [8]. Within the framework of the classification of metacognition processes, B. M. Velichkovsky described five groups of metastrategies [3] while A. V. Karpov defined metacognition as the leading form of reflexive regulation skills [4; 5].

Thus, metacognitive awareness appears to be the main regulating mechanism of cognition and the most-studied phenomenon of metacognition in Russian and foreign research [8; 10; 11; 15; 17].

Research on metacognitive awareness in learning shows its predictive power in relation to the success of learning. In particular, students with a high level of metacognitive awareness are more successful in problem-based learning [19] and gaining expert knowledge [14], and have better results in terms of academic achievement [2; 6; 26; 27; 28; 31; 32]. However, positive correlations between general metacognitive skills and academic achievement are not as strong as might be expected based on theory. The insufficiently strong correlations can be explained in several ways: first, the distribution of the general population, in which respondents with low levels of metacognitive skills are located on both sides of the achievement scale [31]; second, intermediate variables and/or background factors [16; 13]; third, the specifics of the technics used [18].

Foreign researchers have investigated the advantages and disadvantages of metacognition questionnaires and concluded that the questionnaires are valuable for practical and large-scale use, but their structure needs improvement [24]. Thus, despite the fact that metacognition has been thoroughly researched, the question of with what instruments to measure it remains controversial.

The most popular questionnaire among many existing instruments for the assessment of metacognitive components is the Metacognitive Awareness Inventory (MAI) created by G. Shrow and R. Dennison to measure knowledge of cognition and the regulation of cognition [15; 17; 25]. The authors proposed three different options for calculating the subscale scores on the questionnaire: (1) an empirical two-dimensional model, (2) a theoretical two-dimensional model, and (3) an eight-dimensional model (Table 1). Later, researchers began to use a unidimensional model to calculate the total score of metacognitive awareness.

Table 1

Metacognitive Awareness Inventory and the distribution of items between scales

Note: KG = knowledge of cognition; RK = regulation of cognition; DK = declarative knowledge; PK = procedural knowledge; СK = conditional knowledge; P = planning; IMS = information management strategies; M = monitoring; DS = debugging strategies; E = evaluation.

The authors who implement the English-language version of the questionnaire actively (and independently of each other) use four options for processing the questionnaire: the empirical two-dimensional model of the knowledge of cognition (25 items) and regulation of cognition (27 items) [27]; the theoretical two-dimensional model of the knowledge of cognition (17 items) and regulation of cognition (35 items) [33]; the eight-dimensional model [22; 30]; and a unidimensional total model of metacognitive awareness [29].

The Russian-language version of the questionnaire, adapted from English by A. V. Karpov and I. M. Skityaeva, has a unidimensional-factor structure for assessing the total score of metacognitive awareness [5]. However, we identified an eight-factor structure in the questionnaire based on the exploratory factorial analysis of empirical data [1; 2].

The purpose of this study is to clarify the factorial structure of the Russian-language version of the MAI. The following questions guided this research:

1.        Which of the four scoring models used with the MAI (the unidimensional model, the empirical two-dimensional model, the theoretical two-dimensional model, or the eight-dimensional model), is the best in terms of explaining the pattern of responses?

2.        What are the indices of test discrimination and internal consistency of the MAI?

3.        What is an optimal set of items for a short version of the MAI and what fit estimates does it have?

Methodology

In this study, we used the MAI developed by G. Shrow and R. Dennison, adapted by A. V. Karpov and I. M. Skityaeva into Russian.  Each of the questionnaire items is assessed by a respondent using a 5-point Likert scale from “strongly disagree” to “strongly agree” [5].

Participants and data collection. The sample consisted of 527 respondents (136 men) aged from 18 to 39 (M = 23.8 ± 8.8); of which 366 were students in a Bachelor program (students getting their first diploma — SGFD) (M = 19.6 ± 1.33) and 161 were students getting their second diploma (SGSD) (M = 33.4±5.5) at the psychology department of St. Petersburg State University.

Data analysis. To assess the explanatory power of different models for the set of empirical data from Russia, we did a confirmatory factor analysis (CFA) using the maximum likelihood restricted [23].  We used the following indicators of a good model fit: Comparative Fit Index (CFI), Tucker–Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), Akaike Information Criteria (AIC) and Bayesian Information Criteria (BIC) [12; 20]. Correlation analysis for the latent variables of the questionnaire was conducted using Person Correlation Coefficient. We applied Item Response Theory (IRT) analysis to the eight-dimensional model of the MAI, which included analyses of both the outfit and infit statistics of Mean-Square Fit Statistics (MNSQ) and items correlating with the score on the scale [21]. The internal consistency of a set of scale items was assessed with Cronbach’s alpha. We used the students’ t-tests for independent groups to evaluate differences between male and female groups and the two groups of students.

Statistical data processing was carried out using STATA.15 and Winsteps software.

Results of the Study

We analysed four models of the factor structure of the MAI (Table 2) on the whole sample.

Table 2

Fit estimates of the Metacognitive Awareness Inventory’s scoring models

Note: df — the degrees of freedom for chi-square; CFI — Comparative Fit Index; TLI — Tucker-Lewis Index; RMSEA — Root Mean Square Error of Approximation; AIC — Akaike information criteria; BIC — Bayesian information criteria.

The CFI and TLI estimates of all four models are below acceptable limits. It should be noted that the latent variables strongly correlated with each other in the models presented: the empirical model (r = 0.897; p < 0.001) and the theoretical model (r = 0.922; p < 0.001). The correlation coefficients of the latent variables of the eight-factor model ranged from 0.666 to 0.818 at a significance level of all p < 0.001.

Three items — 42, 43, and 52 — were found to have relatively low factor loadings for all four models (Table 1). These particular items add little information in terms of measuring the construct of metacognitive awareness, so we deleted them from subsequent analysis for reasons of maintaining high internal consistency.

IRT analysis of the Metacognitive Awareness Inventory

The analysis of the dimensionality of each of the eight factors of the questionnaire showed that all subscales had a one-dimensional structure. MNSQ values for all items were between 1.5 and 1.9 (all values are less than 2). IRT analysis results determined the average item difficulty of the questionnaire for the whole sample and identified that 18 out of 49 items were problematic: Nos. 32, 5, 46, 3, 15, 22, 45, 37, 41, 31, 39, 9, 47, 48, 30, 21, 19, and 38 (Table 3). The highlighted items disrupted the consistency of the subscales; therefore, they were excluded from further analysis.

Table 3

Fit statistics for the Metacognitive Awareness Inventory (IRT)

Note: In this table, the items with inadequate fit statistics values are highlighted. MNSQ — Mean-Square Fit Statistic; INFIT — Inlier-Pattern-Sensitive Fit Statistic; OUTFIT — Outlier-Sensitive Fit Statistic.

Analysis of the factor structure of the short version of the Metacognitive Awareness Inventory

Based on the results of the IRT analysis, a short version of the questionnaire with 32 statements was comprised. CFA was conducted four times to assess the fit of the four models and clarify the scale structure of the shortened version of the MAI (Table 4).

Table 4

Fit estimates of scoring models of the short version of the Metacognitive Awareness Inventory

Note: df — the degrees of freedom for the chi-square; CFI — Comparative Fit Index; TLI — Tucker–Lewis Index; RMSEA — Root Mean Square Error of Approximation; AIC — Akaike information criteria; BIC — Bayesian information criteria.

In the short version of the questionnaire, the multiple model-fitting criteria improved significantly for all four models. The theoretical two-dimensional model and the eight-dimensional model turned out to be the most accurate, as with the full version of the questionnaire. In addition to the RMSE, the values of the output weights the RMSE of our proposed 32-items instrument were inside the normal range, however, CFI and TLI are slightly below the norm.

The values of Cronbach’s alpha for the unidimensional and two-dimensional models are acceptable for psychological questionnaires (Table 5). Cronbach’s alpha values for the eight-dimensional model are below the lower limit of the acceptable range.

Table 5

The internal consistency of a set of scale items for scoring models of the short version of Metacognitive Awareness Inventory 

Gender and age differences

In the short version of the MAI, the means did not differ significantly in the male and female groups. However, in terms of age, students getting their second degree (M = 54.6±8.8 and M = 45.2±8.4) had significantly higher scores of knowledge of cognition, compared to students in the Bachelor program (M = 52±10.3 and M = 42.9±8.9) in both the empirical two-dimensional model (t(525) = -2.69; p = 0.007) and the theoretical two-dimensional model (t(525) = -3.02; p = 0.003). Also, in comparison with students getting their first diploma, students getting their second diploma had significantly higher scores in declarative knowledge (M = 19.1±3.6 — for SGSD; M = 18±3.9 — for SGFD; t(525) = -3.42; p = 0.001), conditional knowledge (M = 15.3±3.1 — for SGSD; M = 14.5±3.2 — for SGFD; t(525) = 2.81; p = 0.005) and evaluation ( M = 15.9 ± 3.1 for SGSD, M = 14.9 ± 3.3 for SGFD, t(525) = -3.54, p = 0.0004).

Conclusion

The aim of the study was to clarify the factor structure of the Russian version of the MAI questionnaire adapted from English by A. V. Karpov and I. M. Skityaeva. The CFA and IRT analysis made it possible to exclude some of the items from the questionnaire in order to improve the indices of discrimination test for the MAI. The short version of the questionnaire has improved fit estimates for the scoring models. The resultant 32-item structure of the questionnaire is consistent with the results of our previous research, in which some items (No. 3, 4, 22, 25, 32, 41, 42) were excluded from the questionnaire based on the results of factor analysis [1; 2], and also matches the short English version of the questionnaire by 80% [18].

The results of the internal consistency reliability test of the subscales of the short version of the MAI demonstrate that the theoretical two-dimensional model is the most acceptable for scientific research and practice. The eight-dimensional model can be used with some limitations due to the low reliability of some scales. In conclusion, the Russian version of the MAI can be shortened since it reproduces the original factoral structure and increase internal consistency.

Appendix

The short Russian version of the Metacognitive Awareness Inventory

Instructions

Here are several statements about the process of thinking and solving problems. Consider if the statement is true or false as it generally applies to you.  Please use the following ratings to answer:

1 - strongly disagree

2 - disagree

3 - don’t know

4 - agree

5 - strongly agree

Scoring Guide

1. Byzova V.М., Perikova Е.I., Lovyagina A.E. Metakognitivnaja vkljuchennost’ v sisteme psihicheskoj samoreguljacii studentov [Metacognitive Awareness in the System of Students Mental Self-Regulation]. Sibirskiy Psikhologicheskiy Zhurnal [Siberian journal of psychology], 2019, no. 73, рр. 126—140. DOI:10.17223/17267080/73/8 (In Russ.).
2. Byzova V.М., Perikova Е.I., Lovyagina A.E. Jeffektivnost’ metakognitivnyh strategij prinjatija reshenij v uchebnoj dejatel’nosti [Metacognitive strategies of decision making in educational activities: Efficiency in higher education]. Science for Education Today, 2019. Vol. 9, no. 4, рр. 19—35. DOI:10.15293/2658-6762.1904.02 (In Russ.).
3. Velichkovskii B.M. Kognitivnaya nauka: Osnovy psikhologii poznaniya [Cognitive Science: Fundamentals of Cognitive Psychology]. Moscow: Akademiya, 2006. 432 p. (In Russ.).
4. Karpov A.V., Karpov А.А., Karabuschenko N.B., Ivashchenko A.V. Dinamika metakognitivny`x determinant upravlencheskoj deyatel`nosti v processe professionalizacii [Dynamics of metacognitive determinants of management activity in the process of professionalization]. Eksperimental’naâ psihologiâ = Experimental Psychology, 2018. Vol. 11, no. 1, pp. 49—60. DOI:10.17759/exppsy.2018110103 (In Russ.).
5. Karpov A.V. Skityaeva I.M. Psihologiya metakognitivnyh processov [Psychology of metacognitive processes]. Moscow: RAS, 2005. 352 p. (In Russ.).
6. Osorina M.V., Scherbakova O.V., Avanesyan M.O. Problemy` metakognitivnoj regulyacii: normativny`e trebovaniya i neproduktivny`e patterny` intellektual`noj deyatel`nosti [The problems of metacognitive regulation: normative standards and invalid patterns of intellectual performance]. Vestnik Sankt-Peterburgskogo universiteta. Seriya 12. Sociologiya [Vestnik of St. Petersburg State University]. Series 12. Sociology, 2011, no. 2. pp. 32—43. (In Russ.).
7. Falikman M.V. Novaya volna Vy`gotskogo v kognitivnoj nauke: razum kak nezavershenny`j proekt [Elektronnyi resurs] [New Vygotskian wave in cognitive science: The mind as an unfinished project] [Psikhologicheskie Issledovaniya Psikhologicheskie Issledovaniya], 2017. Vol. 10, no. 54. URL: http://psystudy.ru/index.php/num/2017v10n54/1449-%20falikman54.html (Accessed 25.11.2019). (In Russ.).
8. Holodnaya M.A. Psihologiya intellekta: paradoksy issledovaniya [Psychology of Intelligence: Research Paradoxes]. Saint Petersburg: Piter, 2002. 272 p. (In Russ.).
9. Chernokova T.E. Dialekticheskie struktury` v metapoznanii [Dialectical structures in metacognition]. Filologiya i kul`tura [Philology and Culture], 2013. Vol. 33, no. 3, pp. 322—328. (In Russ.).
10. Chernokova T.E. Metakognitivnaya psixologiya: problema predmeta issledovaniya [Metacognitive psychology: the problem of the subject of research]. Vestnik Severnogo (Arkticheskogo) federal`nogo universiteta. Seriya: Gumanitarny`e i social`ny`e nauki [Bulletin of the Northern (Arctic) Federal University. Series: Humanities and Social Sciences], 2011, no. 3, pp. 153—158. (In Russ.).
11. Chartier D., Loarer E. (1997) Obuchenie i perenos kognitivnykh i metakognitivnykh strategiy [Learning and transfer of cognitive and metacognitive strategies]. In Galkina T. (eds.). Kognitivnoe obuchenie: sovremennoe sostoyanie i perspektivy [Cognitive Learning: Current Status and Prospects]. Moscow: Institute of Psychology Publ, RAS, pp. 201—216. (In Russ.).
12. Bartholomew D.J. Steele F., Moustaki I., Galbraith J. Analysis of Multivariate Social Science Data. London: Routledge, 2008. 384 p.
13. Berger J.-L., Karabenick S.A. Construct Validity of Self-Reported Metacognitive Learning Strategies. Educational assessment, 2016. Vol. 21, no. 1, pp. 19—33. DOI:10.1080/10627197.2015.1127751
14. Bransford J.D., Brown A.L., Cocking R.R. How people learn: Brain, mind, experience and school. Washington, D.C: National Academy Press, 2000. 384 p.
15. Brown A.L. Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In Weinert F., Kluwe R. (eds), Metacognition, motivation, and understanding. Mahwah, NJ: Erlbaum, 1987, pр. 65—116.
16. Dent A.L., Koenka, A.C. The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 2015. Vol. 28, no. 3, рр. 1—50. DOI:10.1007/s10648-015-9320-8
17. Flavell J.H. Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 1979. Vol. 34, рр. 906—911. DOI:10.1037/0003066X.34.10.906
18. Harrison G.M., Vallin L.M. Evaluating the metacognitive awareness inventory using empirical factor-structure evidence. Metacognition and Learning, 2018. Vol. 13, no. 1, pp. 15—38. DOI:10.1007/s11409-017-9176-z
19. Hmelo-Silver C.E. Problem-based learning: What and how do students learn? Educational Psychology Review, 2004. Vol. 16, рр. 235—266. DOI:10.1023/B:EDPR.0000034022.16470.f3
20. Kline R.B. Principles and Practice of Structural Equation Modeling. 3rd Edition ed. New York.: The Guilford Press, 2011. 427 p.
21. Linacre J.M. A User’s Guide to WINSTEPS: ProgramManual 3.71.0. [Elektronnyi resurs]. Winsteps. 2011. URL:http://www.winsteps.com/a/winsteps.pdf (Accessed 11.11.2019).
22. Magno C. The role of metacognitive skills in developing critical thinking. Metacognition and Learning, 2010. Vol. 5, no. 2, рр. 137—156. DOI:10.1007/s11409-010-9054-4
23. Rhemtulla M., Brosseau-Liard P.E., Savalei V. When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological methods. 2012. Vol. 17, no. 3, pp. 354—373. DOI:10.1037/a0029315
24. Schellings G., Van Hout-Wolters B. Measuring strategy use with self-report instruments: Theoretical and empirical considerations. Metacognition and Learning, 2011. Vol. 6, no 2, рр. 83— 90. DOI:10.1007/s11409-011-9081-9
25. Schraw G., Dennison R.S. Assessing metacognitive awareness. Contemporary Educational Psychology, 1994. Vol. 19, рр. 460—475. DOI:10.1006/ceps.1994.1033.
26. Schraw G., Moshman D. Metacognitive Theories. Educational Psychology Review, 1995. Vol. 7, no, 4. pp. 351—371. DOI:10.1007/s10648-017-9413-7
27. Sperling R.A. Howard B.C., Staley R., DuBois N. Metacognition and selfregulated learning constructs. Educational Research & Evaluation, 2004. Vol. 10, no. 2, рр. 117—139. DOI:10.1076/edre.10.2.117.27905
28. Tokuhama-Espinosa T. The new science of teaching and learning: Using the best of mind, brain, and education science in the classroom. New York.: Teachers College Press, 2010. 208 р.
29. Turan S., Demirel Ö., Sayek İ. Metacognitive awareness and self-regulated learning skills of medical students in different medical curricula. Medical Teacher, 2009. Vol. 31, no. 10, рр. 477—483. DOI:10.3109/01421590903193521
30. Umino A., Dammeyer J. Effects of a non-instructional prosocial intervention program on children’s metacognition skills and quality of life. International Journal of Educational Research, 2016. Vol. 78, рр. 24—31. DOI:10.1016/j.ijer.2016.05.004
31. Veenman M.V.J., Kok R., Blöte A.W. The relation between intellectual and metacognitive skills in early adolescence. Instructional Science, 2005. Vol. 33, рр. 193—211. DOI:10.1007/s11251-004-2274-8
32. Vrugt A., Oort F. J. Metacognition, achievement goals, study strategies and academic achievement: pathways to achievement. Metacognition and Learning, 2008. Vol. 3, рр. 123—146. DOI:10.1007/s11409-008-9022-4
33. Young A., Fry J.D. Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 2008. Vol. 8 (2), рр. 1—10.
34. Zohar A., Dori Y.J. Metacognition in science education: Trends in current research. Dordrecht: Springer Netherlands, 2012. 280 р. DOI:10.1007/978-94-007-2132-6