Improving the Effectiveness of Educational Quality Management Mechanisms as the Main Condition for Improving School Educational Outcomes

The paper is devoted to improving the results of a general school by introducing changes to the established practices of decision-making and setting development goals that have settled in school management practices. The paper is based on materials of a project of targeted methodological assistance to schools with low educational outcomes, that was implemented from 2020 to 2022 in the Russian Federation. The authors set a goal to identify the existence of a correlation between a number of parameters characterising an educational organisation (the proportion of underachieving students), its management style (the director’s attitudes, the dynamics of changes in management and pedagogical practices) and the likelihood of effective implementation of the development program, observed through increasing students’ outcomes. The results show a straight correlation between the principal's attitudes and the results, a straight connection with different weights of different practices from the study set, and no correlation for the proportion of underachieving students. The conclusion is drawn about the influence of a low level of manager competencies on the director’s attitudes, the feasibility of helping schools in identifying and preventing risk factors, the universality of benefits from the introduction of data-driven management mechanisms data in any school, regardless of the depth of underachievement.

Introduction

The purpose of the concept of "the best school is the one next door" relates to many educational systems of the world. However, in practice, the educational results of schools may significantly differ. In the Russian Federation, according to assessment and evaluation procedures in 2019, more than 9 thousand schools were classified as "schools with low educational results". Classification took place according to the methodology and criteria [1].

Researchers associate the quality gap in schools’ results with various internal and external factors [2]. Positive factors are associated with high outcomes, risk factors, the contrary - with underperformance and underachievement.

The purpose of the Russian "Targeted Methodological Assistance to Schools with Low Educational Results" project (hereinafter referred to as the Project) is to improve the results of schools with poor student performance by identifying the risk factors for reducing the results characteristic of each individual school participating in the project, by developing and implementing measures (development programs) to overcome the negative impact of such factors.

As risk factors, the Project considers risk factors recognized by the scientific community for reducing educational outcomes: contextual constraints, resource shortage and competence deficits, as well as the features of the school educational environment.

Contextual risks are understood as complex factors that the school cannot change itself (for example, the location, social-economic and cultural characteristics of the school’s territory and students background). Contextual risks may also include the characteristics of school students (high socio-cultural diversity, am increased proportion of students with disabilities, etc.). In any education system, such complex factors have a high correlation with lower student outcomes, and students with low socio-economic and cultural status are at risk of low educational outcomes [3].

The other, non-contextual risk factors may be directly affected. For example: the resource shortage (staff, teaching materials and equipment), the quality of the physical environment of the school; the risks of a low-quality school social environment (negative school climate), that affects the quality of social and professional relations at school [4]; low professional competencies of teachers. Researchers often attribute this last category of risk factors to be the most significant [5].

The studies that formed the methodological basis of the Project indicate that contextual risk factors can be mitigated, while the impact of non-contextual factors is significantly reduced or eliminated at all [5, 6, 7].

We will highlight a group of risks associated with school management. The school principal is a key figure in the success of measures to counteract risk factors, planning and implementation is impossible without the direct meaningful participation of the school principal in it. Effective school management today requires the diverse competencies of a school manager: a) assessing deficits and supporting the development of the professional competencies of teachers; b) goal setting, the analysis of the effectiveness of achieving goals, compliance with higher management levels; c) financial and human resources management; d) interaction with the external environment of the school [8].

There at are at least two possible scenarios when the principal and the managerial team do not implement the management function to improve the quality of the results, threating its development: the principal is inactive regarding the identification of risk factors specific to the school, or the principal takes measures which do not lead to the desired results. Inaction can also be the result of little knowledge on how to solve a noticed problem, or a lack of knowledge of the problem, that leads to the inability to make decisions. The situation of taking ineffective measures can also be assumed as a situation of knowledge about the problem, but a lack of skills to solve it.

As a result, the school remains equal to its own basic potential, when the risk factors that threaten its development are contrasted with the professional abilities of the members of the teaching staff, the set of technologies, techniques and practices that have actually developed after receiving pedagogical education and subsequent professional experience, individually implemented by teachers at school. If the skills of the school staff are not enough to mitigate the risk factors, it will demonstrate lower results than if there was an adequate identification, prevention of deficiencies and adaptation of the technologies used to the needs of the school.

The situation of the principal’s inaction regarding the quality of education is quite common in Russian schools. According to the TALIS-2018 study, Russian principals are less involved in issues directly related to improving the quality of education, compared with principals from countries leading in the quality of general education. In a Russian school, the principal pays less attention to identifying teacher deficiencies and adapting the school curriculum (as well as identifying the need for its adaptation) [9].

Not only the lack of necessary competencies, but also the principal’s attitude can interrupt the identification, recognition and elimination of the problems that hinder the development of a school. The lack of successful experience in countering risk factors can support a fixed mindset [10]. The necessity to make steps to change existing practices may be perceived by the principals as an admission of their own failure, which can foster inaction.

Due to the need to change the common inaction of the principal, the main part of the Project involved the creation of the conditions for launching a decision-making process aimed at the school development.

The Project was intended to provide the schools with low education outcomes the following: 1) the information about educational outcomes and the risk factors, and an individual risk profile for a school; 2) a set of selected and approved technologies for developing the practical measures to mitigate the risk factors; 3) methodological advisory support on the development and implementation of measures to mitigate risk factors. Thus, schools received the instruments and instructions for restarting their quality management.

The article discusses the following hypotheses:

• the effectiveness of participation in the Project of Targeted Methodological Assistance to Schools with Low Educational Results correlates with the attitudes of the administrative staff;
• the prevention of low educational outcomes is associated with changes in pedagogical and classroom practices, professional interaction and managerial practices in school;
• the effectiveness of the preventive measures is associated with the starting position of a school receiving assistance (the greater numbers of underperforming students in schools associate with the little efficiency of the assistance).

Method

Comparison of the Results of Subgroups of Schools

The main way to test the hypotheses was to compare the results of different subgroups of schools, identified in the survey results. For this, the following data is used: 1) the lists of schools with low results from different years, the 2019 list is taken as the base, the 2022 list as the control list; 2) the dynamics of the low performance index. The analysis of the dynamics of the index makes it possible to identify differences between effective prevention factors and observe trends in school results even if the school has not left the list of low performers.

The index of low results is calculated for each test of the All-Russian National Tests, separately for grades 4, 5 and 6 in Russian and Math. The index is calculated as the share of the participants in a procedure in a given subject in a given grade, whose results are below the established minimum score value for this grade-subject. The index reflects the proportion of students receiving unsatisfactory grades in the subject. The index takes values from 0–1, where 1 stands for low results, 100% of students in the target grade received “Unsatisfactory” in the subject, 0 means the absence of low results. Since the index characterizes low educational results, its decrease is associated with a decrease in the share of unsuccessful students and an increase of the results. To assess the dynamics, the value of the generalized index of low results is used, equal to the average of all indices of low results according to the specified assessment procedures, including selected grade-subjects results. The base index score is calculated on the results of 2019, the control index is calculated using the same procedure results from 2022.

To analyze the administration's attitudes an index was developed that characterizes school involvement in the Project. It presumes the principal’s view and those school employees that have the greatest influence on his opinion. It includes: the principal's attitude towards the Project, the feedback of the curator (an appointed external practitioner, who is a skilled school manager themselves) collected throughout the Project. The objectives of the survey were the principal's readiness for professional development, their openness to the external environment, their willingness to change the status quo.

The index is calculated using the responses of principals and curators who answered all of the necessary questions during a series of questionnaires (at the beginning, middle, and end of the year of participation in the Project). The number of pairs of a principal and a curator representing one school was 1,590 out of 3,000 pairs who participated in the Project in 2021.

The questionnaires consisted of questions with scoring options (from 1 to 5) and questions with a choice of options. Answers with values 1-3 correspond to a low index value, answers with 4-5 correspond to a high value; all choice options questions used for the index were associated with a decrease in the index (Table 3).

45% of schools showed a high engagement index, 11% showed a low one.

The study of the influence of the change in school practices is made on a sample of underachieving students, the survey of the principals who participated in the Project in 2021 was conducted in 2022. More than 2836 (95%) principals took part in the survey. Survey participants were asked to evaluate how much a particular technology or practice had changed, indicating that it had not changed at all (1 point) or, on the contrary, that it had become developed enough that the school could present its own experience in that area (5 points). The results of the survey were compared for schools that remain in the list of schools with low results since 2019 or had left it in 2022.

To analyze the connection between the initial level of underperformance in school and the results of participation in the Project, the dynamics of the results of the index of low results in the subgroups of schools is analyzed. The following groups are distinguished: basic underachievers (<= .32); average underachievers (.33–.47; .48–.60); severe underachievers (>= .61). The share in the brackets is the low performing students according to All-Russian National Tests. Among the Project participants, the shares of schools with different levels of failure and contextual risks are distributed in equal proportions to the actual state at the beginning of the Project, this was one of the specific conditions for selecting participating schools [11].

The value of the index of low results, the share of public organizations that were on the list with low results and left it are presented in Table 2.

Table 1. Questions Used for the Index of School Involvement in the Implementation of the Targeted Methodological Assistance Project in 2021

Table 2. The Relationship Between the Factors of the Prevention of Low Results, the Fact that a School is Included in the List of Low Performers and the Dynamics of the Low Results Index

Results

As a result of hypothesis testing, it was revealed that the attitudes of the principal are associated with the successful implementation of the Project. Schools whose principals demonstrated high involvement in the implementation of the Project were on average 1.7 times more effective than principals who did not show such involvement.

Increased confidence in the use of classroom technologies and managerial practices that were mentioned in the questionnaire is associated with a reduction in the proportion of low educational results. For example, the most noticeable changes for the principal in the pedagogical technologies used by teachers and the organization of the school system for assessing the quality of education resulted in the most noticeable decline in the index of low results (-,16), as well as the use of educational practices by teachers (-,12); the visiting of each other's lessons by colleagues (-,10); providing feedback to colleagues (-,09).

The hypothesis of a higher difficulty of preventing low results in schools with a higher proportion of underachieving students is not confirmed. On the contrary even, the schools with the largest proportion of students with low results have achieved the most noticeable dynamics of reducing the number of underachieving students. Note, however, that a more favorable starting position turned out to be associated with a more successful exit from the list of low performing schools, probably due to the fact that the results are closer to the boundary of the criteria chosen to determine the list, and, consequently, even a small improvement takes the school out of the focus zone.

As part of the discussion, it should be noted that, as a result of the differences in the evaluation scales of the studied groups of prevention factors (attitudes, practices, initial position), one should interpret with caution the comparison of the strength of the factors described by the change in the index of low results. So, for example, the active position of the principal can become a determining factor for the school to improve results, while the fixed mindset of the impossibility of improving results can completely block the development of a school. At the same time, it seems that, in general, both the trend and the magnitude of the listed factors reflect reality.

The lack of standards and generally accepted criteria for the qualitative implementation of the listed communication and managerial practices significantly complicate the assessment of the effectiveness of their implementation in schools. The more unambiguously interpreted aspects, which were covered within the Project's methodological framework, such as "pedagogical technologies", "organization of IES", "observation of colleagues' lessons", turned out to be associated with more noticeable positive dynamics than more abstract categories such as "quality of communication between teachers and students", "school climate". At the same time, there is good reason to believe that the potential for the development of these aspects of the school's activities is more than significant.

An important factor that can be considered in relation to the activity of a school, but that is not sufficiently covered by the analysis is the external environment – the administrative structure in the school’s founder and the region where school is located. Schools facing inefficient management and schools receiving the necessary assistance obviously work in different conditions. In this regard, the attitude and criteria that municipal and regional managers use for determining schools with low outcomes are very important. For example, an educational organization with high contextual risks and a significant proportion of underachievers may not be considered by the municipality and the regional executive authority as a school in need of support until it gets into the federal list of public organizations with low results. It seems risky for achieving the national goal of improving the quality of education.

Conclusion

To summarize, let us record a number ofconclusions.

• Low school results are a consequence of the conjunction of a number of circumstances: an increased concentration of risk factors, insufficient counteraction or the inaction of the school administration.
• The inaction of the school is associated with the low competence of the administration in identifying and counteracting the risk factors. Inaction could be provoked by the absence of mechanisms for managing the quality of education: schools don’t have enough technologies to identify problems and introduce the appropriate measures to mitigate negative impact.
• The administration's attitudes may influence the implementation of quality management mechanisms for education at school.
• A school regardless of the starting point and the share of underachieving students can improve its results if it begins to implement mechanisms for managing the quality of education. Approaches to managing the quality of educational outcomes at school are universal.

1. Dweck C. Gibkoe soznanie: novyj vzglyad na psihologiyu razvitiya vzroslyh i detej [Flexible consciousness: a new look at the psychology of adult and child development]. Russian language, 7th release, 2021.
2. Metodika vyyavleniya obshcheobrazovatel'nyh organizacij, imeyushchih nizkie obrazovatel'nye rezul'taty obuchayushchihsya, na osnove kompleksnogo analiza dannyh ob obrazovatel'nyh organizaciyah, v tom chisle dannyh o kachestve obrazovaniya [Methodology for identifying general education schools with low educational results of students based on a comprehensive analysis of data on educational organizations, including data on the quality of education]. 2020. Available at: https://fioco.ru/antirisk (Accessed 18.06.2023).
3. Report of the TALIS-2018 national study coordinator. 2018. Vol 1. Available at: https://fioco.ru/Talis-18-results (Accessed 28.07.2023).
4. Pinskaya M.A. et al. Poverh bar'erov: istorii shkol, rabotayushchih v slozhnyh social'nyh usloviyah. Kniga dlya shkol'nyh administratorov i dlya tekh, kto upravlyaet shkolami na municipal'nom i regional'nom urovnyah [Over the barriers: stories of schools operating in difficult social conditions. A book for school administrators and for those who manage schools at the municipal and regional levels]. Higher School of Economics, 2019. Available at: http://publications.hse.ru/mirror/pubs/share/direct/339289224 (Accessed 28.07.2023). (In Russ.).
5. Order Federal Service of Supervision in Sphere of Education and Science from 19.08.2020 No 847. Available at: https://fioco.ru/научно-методические-материалы (Accessed 18.06.2023). (In Russ.).
6. Order Federal Service of Supervision in Sphere of Education and Science from 30.08.2020 No. 871. Metodika okazaniya adresnoj metodicheskoj pomoshchi obshcheobrazovatel'nym organizaciyam, imeyushchim nizkie obrazovatel'nye rezul'taty obuchayushchihsya [Methods of providing targeted methodological assistance to general education schools with low educational results of students]. Available at: https://fioco.ru/antirisk (Accessed 18.06.2023). (In Russ.).
7. Agasisti T. et al. Academic resilience: What schools and countries do to help disadvantaged students succeed in PISA. OECD Education Working Papers, 2018, no. 167. DOI:10.1787/e22490ac-en
8. Echazarra A., Radinger T. Learning in rural schools: Insights from PISA, TALIS and the literature. OECD Education Working Paper, 2019, no. 196. URL: https://one.oecd.org/document/EDU/WKP(2019)4/En/pdf (Accessed 28.07.2023).
9. Gase L. et al. Relationships among student, staff, and administrative measures of school climate and student health and academic outcomes. Journal of School Health, 2017. Vol. 87/5. DOI:10.1111/josh.12501
10. Pont B. et al. Improving School Leadership. OECD Policy and Practice. 2008. Available at: https://www.oecd.org/education/school/Improving-school-leadership.pdf (Accessed 18.06.2023).
11. Schleicher A. World Class: How to Build a 21st-Century School System. Strong Performers and Successful Reformers in Education. OECD Publishing, Paris, 2018. DOI:10.1787/9789264300002-en