Educational and Motivational Predictors of Academic Achievement (Based on PISA 2018 Reading Data)

Academic achievements of teenage students are an important indicator of their further success and adaptation to life in adult society. The material of the study was the data collected for the international project PISA 2018 on a representative sample of Russian teenagers (N=7608). The article presents the results of studying the role of educational and motivational factors (controlling for gender and family environment) of academic achievements of Russian teenagers based on the material of PISA 2018 on reading literacy. We confirmed that regarding the environmental (family and teacher) factors in the academic achievements of schoolchildren, the role of SES as an important predictor of schoolchildren's academic achievements, the role of teacher support for active involvement in reading is significantly positive factor, and Teacher-directed instruction is negative factor. The study confirmed an important contribution of motivational variables to reading literacy, reading engagement was proved to be a positive predictor, and fixed mindset about intelligence was proved to be a negative predictor of reading literacy competence. The discussion shows that the data obtained generally corresponds to international data on predictors of academic achievement among schoolchildren based on the PISA 2018. The results obtained can be used in the context of teacher training and for improving the quality of education in Russian schools.

Introduction

The international PISA studies, conducted since 2000, make it possible to assess the quality of education in different countries and take measures to improve it. Assessment of students' educational achievements is carried out in three main areas - reading literacy, mathematical literacy and natural science literacy. The study is carried out in three-year cycles, with each cycle focusing on one of the three areas indicated above. The relevance of this study is due to the importance of analyzing the factors behind the achievements of schoolchildren and the possible specificity of these factors for each participating country [4; 12]. At the family level, the only universal variable showing positive associations with success in PISA tests is the family's socioeconomic and cultural status (SES) [8; 9] (even though discussions about approaches to its assessment continue (see [7])).

77 countries took part in the PISA 2018 study. Russian teenagers showed results higher than the average for all participating countries (453 points), but slightly lower than the OECD average. At the same time, the results shown by Moscow schoolchildren were third in the world, after China (4 provinces) and Singapore. Compared to previous measurements from 2000-2012, there was a continuation of the positive dynamics of the results of Russian schoolchildren.

Since 2009, indicators of the quality of education have been included in the PISA diagnostic battery for schoolchildren. These indicators are constantly expanded and refined. There are studies on the  role of disciplinary climate in the classroom [4; 14; 16; 19]. Disciplinary climate was found to explain 11% of interschool variation in reading achievement across countries [14]. However, in 12 of 65 countries no connections were found, which may indicate a certain ambiguity of this educational strategy, which is part of an integral educational system with its priorities, values, and goals. Similar inconsistent findings were found for the teacher support dimension [16].

Another teaching style variable that has been extensively studied in PISA is teacher-directed instruction. It involves technologies in which the teacher is the primary agent of learning, as opposed to student-mediated learning in which students take more responsibility for their own and peers' learning. The results of research obtained as part of the PISA 2015 project on the impact of directive instruction, assessed as the teacher’s ability to explain scientific ideas, on educational results in natural science indicate its effectiveness [5]. However, the data obtained from the analysis of the PISA 2018 results [12; 16], generally do not confirm this result. The analysis shows that the conflicting results on directive learning are related to its specific operationalization in different studies (in particular, PISA 2015 and PISA 2018). It could be also to the fact that teacher directed instruction can have different consequences in classes with different levels of preparedness: for example, it can be used deliberately in classes with low achievements in order to adapt instruction to the level of students’ preparation.

The role of another pedagogical factor – the teacher’s stimulation of reading engagement was studied using data from 12 thousand schoolchildren from three countries – Turkey, China and Mexico [12]. It was shown that in all three countries this indicator made a positive contribution to the academic achievements of schoolchildren, but the extent of this contribution varied: it was most significant for Chinese students and least significant for Mexican students.

The study of the role of psychological variables as predictors of achievement in PISA tests indicates the contribution of two main factors: intrinsic motivation and its analogues [3; 17] and academic self-efficacy [13]. This generally corresponds to the data of recent meta-analyses dedicated to the association of various motivational and personal factors to students’ academic achievements [10; 18].

This article attempts to study the role of educational and motivational factors (controlling for gender and characteristics of the family environment) in the academic achievements of Russian adolescents using the PISA 2018 reading literacy test. Reading literacy is described as an understanding, using, evaluating, reflecting on and engaging with texts in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society [15]. Based on the literature review, specific hypotheses were put forward regarding two types of educational factors (teachers’ behavior and family support) and two main motivational factors. First, we hypothesized that teacher-directed instruction, which frustrates students' need for autonomy, would contribute negatively to reading literacy achievement, while teacher's stimulation of reading engagement perceived by student would contribute positively to PISA scores. Secondly, we hypothesized that enjoyment of reading (analogue of intrinsic motivation) would be the most important positive predictor of reading achievement, and that the fixed mindset (in accordance with C. Dweck’s theory) would be the most important negative motivational predictor of reading achievement.

Method

Sample. This study used a representative sample of students from the PISA 2018, including 7.608 15-year-olds from 263 educational organizations in 43 regions of Russia. The sample includes 3.861 (50.7%) girls and 3.747 (49.3%) boys.

Measures. The data used in the analysis was obtained from PISA 2018 using measurements developed by a consortium of organizations specifically for PISA’s goals. They  include test items to assess reading literacy, as well as a number of scales assessing certain characteristics of students and the learning environment. Reliability and validity of these scales were shown by the organizers PISA [15]. To make the most of PISA data, when selecting variables for analysis of reading scores predictors, we included all available personal and motivational characteristics of students, as well as characteristics of their perception of the school environment and the teacher. We also used in our analysis:

1. gender (0 – female, 1 – male);
2. immigrant background of the family (1 – native resident, 2 – first- or second-generation immigrant);
3. the language used at home most of the time (1 – Russian, 2 – other);
4. skipping classes or days of school (if responding to the question “In the last two full weeks of school, how often:” the student assessed the items “I skipped a whole school day” and “I skipped some classes”, choosing the option “Never”, then he received 0, in all other cases — 1);
5. arriving late for school (if responding to the same question the student rated the item “I arrived late for school” by choosing the option “Never”, then s/he received 0, otherwise – 1);
6. index of economic, social and cultural status. SES estimates, presented by PISA organizers in the form of a standardized quantitative index, are made up of three other indicators with equal weights: maximum parental level of education, maximum professional status of parents and items available in the home, including books (for more information on the composition and calculation of the SES index see [16], pp. 216-217]).

We considered the following characteristics of the educational and family environment as perceived by a teenager among the possible predictors of reading literacy scores.

1. Characteristics of teacher behavior:
   • Teacher's stimulation of reading engagement (4 items, for example, “The teacher encourages students to express their opinion about a text”);
   • Disciplinary climate (5 items, for example, “Students don’t listen to what the teacher says”);
   • Teacher-directed instruction (4 items, for example, “The teacher sets clear goals for our learning”);
   • Teacher support (4 items, for example, “The teacher gives extra help when students need it”);
   • Adaptive instruction (3 items, for example, “The teacher adapts the lesson to my class’s needs and knowledge”);
   • Teacher enthusiasm (4 items, for example, “The enthusiasm of the teacher inspired me”);
   • Teacher feedback (3 items, e.g., “The teacher gives me feedback on my strengths in this subject”).
2. Characteristics of the school environment:
   • Perception of competitiveness at school (3 items, for example, “Students seem to value competition”);
   • Perception of cooperation at school (3 items, e.g., “Students seem to value co-operation”);
   • Exposure to bullying (6 items, e.g., “I was threatened by other students”).
3. Parents' emotional support (3 items, e.g., “My parents support me when I am facing difficulties at school”).

Also, a number of motivational characteristics were considered among the likely predictors of reading achievements:

• Enjoyment of reading (5 items, for example, “Reading is one of my favorite hobbies”);
• Fixed mindset (1 item: “Your intelligence is something about you that you can't change very much”);
• Mastery goal orientation (3 items, e.g., “My goal is to completely master the material presented in my classes”);
• Achievement motivation (3 items, e.g., “I find satisfaction in working as hard as I can”);
• General self-efficacy (5 items, e.g., “I usually manage one way or another”);
• Fear of failure (3 items, e.g., “When I am failing, I worry about what others think of me”);
• Competitiveness (3 items, e.g., “I enjoy working in situations involving competition with others”);
• Value of school (3 items, e.g., “Trying hard at school will help me get a good job”).

A four-point response scale is used in each of the measures listed above, with the exception of mastery goal orientation, where the response scale includes five gradations. Scores for these measures were calculated by the PISA organizers using the two-parameter IRT model and standardized based on a general sample from all OECD countries [16, p. 212].

Data analysis methods. To assess the impact of each variable to PISA reading literacy scores regression analysis (RA) was carried out in the Mplus 8 program using the maximum likelihood method with a robust estimate of standard errors (MLR) while considering replication weights (taking them into account provides a more accurate estimate of the standard errors of the parameters, but does not allow determine the fit indexes of the model). During RA and structural equation modeling we used Mplus option for multiple imputation analysis, which was recommended by organizers of PISA to calculate 10 possible reading literacy scores. The RA was carried out step by step: at the first step, the basic indicators describing the child and his family were included in the model, a set of variables describing the student’s perception of the school environment, teacher and parents was added to the next model, and at the last step a set of motivational variables was added to the model.

To analyze the possible mediating role of motivational factors, structural equation modeling was carried out in Mplus 8 using the MLR method and cluster design (considering the distribution of students by school), which allows to obtain unbiased estimates of standard errors [11]. To be able to assess the fit of the model to the data, we used full weights instead of replication weights during structural modeling. The statistical significance of mediated effects was assessed using bootstrap analysis (5000 samples).

Results

The results of the RA (see Table) demonstrate that in the first model, SES shows the greatest positive relation to the reading literacy scores. Gender is also associated with scores: they are slightly lower for boys. Reading literacy is related to the language used at home most of the time: if it differs from Russian, then the grades are lower. Skipping classes or days of school and arriving late for school showed negative relations with reading literacy.

After adding indicators of perceived school environment and parental support to the model (Model 2 in the table), all variables considered (with the exception of skipping classes or days of school) continue to show statistically significant associations with scores and SES remains the strongest predictor. Among the indicators of the school environment, the strongest positive relationship is demonstrated by teacher's stimulation of reading engagement and adaptive instruction. Parents' emotional support and the perception of cooperation at school also show a positive association to PISA scores. Reading literacy was found to be negatively related to teacher-directed instruction, teacher feedback, perception of competitiveness at school, and exposure to bullying.

Table. The results of a regression analysis for PISA reading literacy score

Note. * – p≤0.05; ** – p≤0.01; *** – p≤0.001.

In the third model, the regression coefficient of gender becomes insignificant, which may indicate that the relationship between scores and gender is mediated by some motivational factors. Other predictors remain significant, although the strength of the relationship decreases: this is especially pronounced in relation to SES. The largest positive relationship with reading literacy demonstrates enjoyment of reading. A smaller positive association was found for SES, teacher's stimulation of reading engagement, adaptive instruction, parents' emotional support, perception of cooperation at school, and of the motivational variables — competitiveness and fear of failure. Reading literacy scores were negatively related to the language used at home most of the time, arriving late for school, teacher-directed instruction, teacher feedback, exposure to bullying, perception of competitiveness at school, and among the motivational variables —fixed mindset and mastery goal orientation.

A comparison of the explained variance proportion in each of the models demonstrates a more important role of motivational factors and family characteristics, while the impact of the perceived school environment and teacher is somewhat weaker. The final model explains 26% of the variance in scores, which is approximately the same as the proportion of between-student reading score variance explained by similar factors in the PISA 2009 study using Russian sample, albeit with a smaller set of variables [2].

To analyze the mediating effects of motivational factors we compiled a structural model, including as the main predictors the variables with the largest regression coefficients: enjoyment of reading and SES. Of the characteristics of the school environment, teacher's stimulation of reading engagement and teacher-directed instruction were included in the model with covariation between them. We also included in the model gender which showed a varied impact on the reading literacy scores across RA models, suggesting the possibility of an indirect effect. Enjoyment of reading was hypothesized to partially mediate the effect of other variables on reading literacy scores. Evaluation of this model (see figure) demonstrated an excellent fit to the data: χ2=114.07; df=9; CFI=0.953; RMSEA=0.040 (N=7139). This model explains 19% of the variance in reading literacy scores.

Figure. A structural model of the relations between reading literacy scores and main motivational, educational and family predictors (all coefficients are standardized and statistically significant at p<0.001, with the exception of the coefficient from teacher-directed instruction to enjoyment of reading)

Among the mediated effects, the following are statistically significant:  gender (-0.06; p≤0.001), socio-economic status (0.03; p≤0.001), teacher's stimulation of reading engagement (0.04; p≤0.001) and fixed mindset (-0.02; p≤0.001).

Discussion

Students’ academic achievement is the main educational outcome that modern school systems around the world strive for. Educational achievement in reading is important not only in itself, but also because it is a strong predictor of student achievement in mathematics and science. The results of the analysis confirmed the role of SES, which was previously shown among the important predictors of schoolchildren’s academic achievements [2], even though in Russia the contribution of SES is relatively small, and SES itself is quite high.

Of particular interest is a group of educational factors that can be influenced by adjusting teacher training and orienting them towards more effective strategies and teaching styles. In relation to educational variables, the role of teacher support for reading engagement was shown both for adolescents' own interest in reading and in their reading literacy. The negative impact of teacher directed instruction on reading literacy has also been demonstrated. These data correspond to international ones [12] and can be meaningfully interpreted from the framework of self-determination theory, currently the most well-known approach to understanding the sources of intrinsic motivation. Thus, teacher-directed instruction perceived by students is an attempt by the teacher to control the entire course of learning, independently regulating the actions of children in a rather directive, non-negotiable manner, which frustrates the students’ need for autonomy. The consequences of such frustration typically include decreased autonomous motivation, persistence, and engagement in the learning process, which likely explains the negative effect of directive instruction on reading achievement. However, the results indicate that teacher directed instruction does not have a direct effect on reading enjoyment), reflecting a distinction between learning motivation and the more specific reading motivation that depends on factors other than the teacher's. Even the teacher’s use of special reading stimulation techniques, as can be seen from the presented model, has only a very limited effect on reading enjoyment, comparable in magnitude to the SES effect and noticeably smaller than the gender effect. The relatively weak dependence of enjoyment of reading, the most important motivational predictor of PISA achievements in reading literacy, on the educational environment convinces of the need for further analysis of the factors that determine it, which constitutes the prospect of this study.

The strategy for stimulating reading engagement involves teacher support for the students’ intellectual activity, their involvement in the process of studying literary texts, directing them to search for connections between the material being studied and their own lives, and discussions about the material under discussion. This kind of activity can support another basic psychological need - the need for competence, which has a positive effect on interest and motivation for reading. The resulting model demonstrates that reading stimulation does support reading engagement, but the indirect effect on reading achievement is only partial to the direct effect. In other words, reading stimulation techniques support students’ activity, leading to positive results even in cases where it is not accompanied by an increase in reading enthusiasm. At the same time, the use of reading stimulation to increase the activity of students may not lead to an increase in engagement if stimulation techniques are used in a controlling manner, when, despite inclusion in the appropriate activity, conditions for satisfying the needs for competence and autonomy do not arise. The possibility of such a controlled use of reading stimulation techniques is also evidenced by the fairly close direct connection between reading stimulation and directive teaching, which is typical not only for the Russian education system, but also for schools in OECD countries [16]. The frequent use of reading stimulation techniques in a controlling style can also explain the rather moderate magnitude of their effect on reading enjoyment.

The fundamental role of motivational variables and, in particular, joy for reading in reading literacy was discovered, which confirms earlier data obtained on a Russian sample [2]. A new and important result discovered using PISA 2018 data shows the role of the fixed mindset (entity theory of intelligence) as a negative predictor of academic achievements of Russian schoolchildren and the mediation of this contribution by enjoyment of reading. This result is of particular interest considering the recent debate about the predictive value of entity and incremental implicit theories of intelligence (see meta-analysis [20]) and their possible cultural specificity [6].

The results obtained are of interest both from a theoretical and practical points of view, largely consistent with the data of other analyzes conducted within the framework of the PISA project [2; 12], as well as psychological and pedagogical research aimed at searching for educational and psychological sources of academic achievement [1; 10; 18]. At the same time, several data reveal cultural specificity: thus, in contrast to the data obtained on schoolchildren in Turkey, China and Mexico [12], the disciplinary climate turned out to be an insignificant predictor of reading achievements in Russian adolescents.

Conclusions

1. The analyses of the results of the role of educational and motivational factors (controlling for gender and family environment) in the academic achievements in reading literacy of Russian adolescents are presented; they are based on the PISA 2018 data which emphasized reading literacy. The contribution of SES and gender to reading achievement and their relationship with reading motivation is shown: the higher achievements of girls and students from families with higher SES are largely due to their greater motivation for reading (enjoyment of reading).
2. Regarding the contribution of educational factors to reading literacy, the positive role of teacher stimulation of students' active involvement in reading and the negative role of teachers’ directed instruction (which most likely frustrates students’ autonomy and competence needs) are shown.
3. A significant contribution of motivational variables to PISA reading literacy has been shown: namely, adolescents’ enjoyment of reading contributes to the achievement of higher results, while fixed mindset (or entity theory of intelligence according to C. Dweck) is associated with lower achievements.

1. Gordeeva T.O., Sychev O.A. Motivatsionnye profili kak prediktory samoregulyatsii i akademicheskoi uspeshnosti studentov [Motivational profiles as predictors of students’ self-regulation and academic achievement]. Vestnik Moskovskogo Universiteta = Moscow University Psychology Bulletin, 2017, no. 1, pp. 67–87. DOI:10.11621/vsp.2017.01.92 (In Russ.).
2. Kuzmina Yu.V., Tyumeneva Yu.A. Chitatel'skaya gramotnost' 15-letnikh shkol'nikov: znachimost' semeinykh, individual'nykh i shkol'nykh kharakteristik [Reader’s literacy of 15-year-olds: the importance of family, individual and school characteristics (According to the Russian sample PISA 2009)]. Voprosy obrazovaniya = Educational Studies, 2011, no. 3, pp. 164–191. DOI:10.17323/1814-9545-2011-3-164-191 (In Russ.).
3. Akben-Selcuk E. Personality, motivation, and math achievement among Turkish students: Evidence from PISA data. Perceptual and Motor Skills, 2017. Vol. 124, no. 2, pp. 514–530. DOI:10.1177/0031512516686505
4. Berkowitz R., Moore H., Astor R.A., Benbenishty R. A research synthesis of the associations between socioeconomic background, inequality, school climate, and academic achievement. Review of Educational Research, 2017. Vol. 87, no. 2, pp. 425–469. DOI:10.3102/0034654316669821
5. Cairns D., Areepattamannil S. Teacher-Directed Learning Approaches and Science Achievement: Investigating the Importance of Instructional Explanations in Australian Schools. Research in Science Education, 2022. Vol. 52, no. 4, pp. 1171–1185. DOI:10.1007/s11165-021-10002-0
6. Costa A., Faria L. Implicit Theories of Intelligence and Academic Achievement: A Meta-Analytic Review. Frontiers in Psychology, 2018. Vol. 9. DOI:10.3389/fpsyg.2018.00829
7. Eriksson K., Lindvall J., Helenius O., Ryve A. Socioeconomic Status as a Multidimensional Predictor of Student Achievement in 77 Societies. Frontiers in Education, 2021. Vol. 6. DOI:10.3389/feduc.2021.731634
8. Gamazo A., Martínez-Abad F., Olmos-Migueláñez S., Jose Rodriguez-Conde M. Assessment of factors related to school effectiveness in PISA 2015. A multilevel analysis. Revista de Educacion, 2018, no. 379, pp. 56–84. DOI:10.4438/1988-592X-RE-2017-379-369
9. Harwell M., Maeda Y., Bishop K., Xie A. The Surprisingly Modest Relationship Between SES and Educational Achievement. The Journal of Experimental Education, 2017. Vol. 85, no. 2, pp. 197–214. DOI:10.1080/00220973.2015.1123668
10. Howard J.L., Bureau J., Guay F., Chong J.X.Y., Ryan R.M. Student Motivation and Associated Outcomes: A Meta-Analysis From Self-Determination Theory. Perspectives on Psychological Science, 2021. Vol. 16, no. 6, pp. 1300–1323. DOI:10.1177/1745691620966789
11. Hoyle R.H. Handbook of Structural Equation Modeling. New York: Guilford Publications, 2023. 801 p.
12. Koyuncu I., Firat T. Investigating Reading Literacy in PISA 2018 Assessment. International Electronic Journal of Elementary Education, 2020. Vol. 13, no. 2, pp. 263–275. DOI:10.26822/iejee.2021.189
13. Kriegbaum K., Jansen M., Spinath B. Motivation: A predictor of PISA’s mathematical competence beyond intelligence and prior test achievement. Learning and Individual Differences, 2015. Vol. 43, pp. 140–148. DOI:10.1016/j.lindif.2015.08.026
14. Ning B., Van Damme J., Van Den Noortgate W., Yang X., Gielen S. The influence of classroom disciplinary climate of schools on reading achievement: a cross-country comparative study. School Effectiveness and School Improvement, 2015. Vol. 26, no. 4, pp. 586–611. DOI:10.1080/09243453.2015.1025796
15. OECD. PISA 2018 assessment and analytical framework. Paris: OECD publishing, 2019. 308 p. DOI:10.1787/b25efab8-en
16. OECD. PISA 2018 Results (Volume III): What School Life Means for Students’ Lives. Paris: OECD Publishing, 2019. 368 p. DOI:10.1787/acd78851-en
17. Ozel M., Caglak S., Erdogan M. Are affective factors a good predictor of science achievement? Examining the role of affective factors based on PISA 2006. Learning and Individual Differences, 2013. Vol. 24, pp. 73–82. DOI:10.1016/j.lindif.2012.09.006
18. Richardson M., Abraham C., Bond R. Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 2012. Vol. 138, pp. 353–387. DOI:10.1037/a0026838
19. Scherer R. The Case for Good Discipline? Evidence on the Interplay Between Disciplinary Climate, Socioeconomic Status, and Science Achievement from PISA 2015. In: T.S. Frønes et al. (eds.). Equity, Equality and Diversity in the Nordic Model of Education. Cham: Springer International Publishing, 2020, pp. 197–224.
20. Sisk V.F., Burgoyne A.P., Sun J., Butler J.L., Macnamara B.N. To What Extent and Under Which Circumstances Are Growth Mind-Sets Important to Academic Achievement? Two Meta-Analyses. Psychological Science, 2018. Vol. 29, no. 4, pp. 549–571. DOI:10.1177/0956797617739704