Student Academic Competence Questionnaire

The results of the development and standardization of an academic competence questionnaire are presented. The method is based on the understanding of academic competence as a complex of independently developed personal qualities, abilities and skills that ensure the success of educational activities in conditions of its variability during higher education. The theoretical model includes 6 structural and content components (academic motivation, self-organization of educational activities, emotional self-regulation, skills and abilities to work with information, proficiency in oral and written language, communication skills), presented at 3 levels of generality and freedom of transformation (reproductive, reflective, functional). Based on the results of Study 1 (N=199), the preliminary version of the questionnaire was reduced from 144 to 69 items. Comparing the results of successful and unsuccessful students confirmed predictive validity. External criterion validity using the contrast group method is based on comparison of groups of undergraduate and graduate students, correlations with self-organization and academic motivation questionnaires (AMS). Study 2 (N=355) demonstrated a relatively acceptable fit of the data to the theoretical model based on confirmatory factor analysis, consistency with educational experience questionnaire scales. The internal consistency of the scales was confirmed by Cronbach's alpha indicators based on data from studies 1 and 2. Descriptive statistics for Study 2 are provided. Standardization is proposed based on percentile values, because the distributions of values for most scales differ significantly from normal.

Introduction

Despite the development of approaches to the formation of universal learning activities in high school [16], university students are variably capable of independently regulating and conducting their learning activities [15]. Various practices to address this problem are emerging in universities. These may include adaptation training sessions incorporated into the curriculum or organized by the university's psychological services, reliance on the work of curators [4; 20], and others. The effectiveness of these practices can be evaluated using methods aimed at measuring indicators of subjective well-being [2], motivation, academic adaptation [1], student performance, and so on. Such diagnostic procedures do not always provide information about the strengths and weaknesses of a specific student in relation to their learning activities, which is significant for aiding in the reflection and construction of an individual development plan.

At the Faculty of Psychology and Social Work of VGSPU, with the goal of forming instrumental competence in students for conducting learning activities during higher education, the subject "Practicum of Academic Competence" was included in 2011 in the bachelor's degree programs in "Psychological and Pedagogical Education" and "Psychology" [8]. The content and forms of organizing the academic work within the practicum were developed based on the ideas of the theory of developmental learning, the methodology of cultural-historical psychology, and the activity approach, which is reflected in the developed course guide [9]. Based on these developments, a similar course was implemented at Yanka Kupala State University of Grodno (Belarus) [11]. The practicum involves various diagnostic tools in the mode of reflexive self-diagnosis. Based on the success of this experience and the need for a systematic assessment of the developmental effects of the practicum, the goal was set to develop a questionnaire of academic competence.

Previously, academic competence (AC) was defined as a complex of independently developed personal qualities, abilities, and skills ensuring the success of learning activities in their variability during higher education [10], which can be understood as a system of specific resources of learning activities significant for the development of personal potential, based on the works of D.A. Leontiev [6].

From a content perspective, the structure of AC was defined based on the conceptualization of the practicum implementation experience; however, its content similarity to the model proposed by S. Elliott and D.K. DiPerna for schoolchildren and used in contemporary research [19], and its consistency with studies on predictors of student learning success [14; 18], can be noted. We also consider the possible formation of AC at different levels in terms of generalization and freedom of its transformation in changing conditions, based on the concept of functional development as a process where "three stages (or levels) of the formation of cultural action methods can be identified" [13, p. 31]. Thus, at the formal or reproductive level, AC manifests in spontaneously formed typical behavior patterns and learning activity methods that ensure its effectiveness. The reflexive level ensures the awareness of choosing a method of activity, guided by the requirements of the situation and one's own capabilities. The functional level corresponds to the ability to transform one's learning activity methods based on reflexive analysis of the current situation, using various cultural tools.

The structural-content model of AC includes the following components: (1) academic motivation – as a characteristic of overall motivation for learning and the ability to recognize and enhance it; (2) self-organization of learning activities ensures the success of planning and implementing all types of academic work; (3) emotional self-regulation supports the stability and positivity of emotional states in learning; (4) skills and abilities to work with information are considered in the context of its search and quality assessment, as well as its transformation and understanding; (5) proficiency in oral and written language ensures the success of academic writing and oral presentations; (6) communication skills and abilities are important for successful communication in the educational process. Thus, the generalized model of AC includes 6 specified components, which can manifest differently at reproductive, reflexive, and functional levels.

Methods and organization of the study

In developing the AC questionnaire according to the theoretical model, 6 substantive scales were included, each with 3 subscales corresponding to the levels of its formation, totaling 18 subscales. For each, 8 items were formulated, resulting in a preliminary version of 144 statements for dichotomous evaluation. The statements were discussed with a group of 4 experts – master's students of the "Cultural-Historical Psychology and Activity Approach in Education" program (one of whose research was related to studying academic competence [12], while the others were then university psychology lecturers with over 15 years of experience). All statements were deemed consistent with the theoretical model of academic competence.

Data collection for the empirical study 1 was conducted from 2020 to 2022, with the pilot version of the developed questionnaire included in the program alongside the following methodologies to assess its validity:

Questionnaire "Academic Motivation Scales" (AMS) (T.O. Gordeeva, O.A. Sychev, E.N. Osin), based on theoretical representations of intrinsic and extrinsic motivation in self-determination theory [3]. The questionnaire includes 7 scales diagnosing three types of intrinsic motivation (cognitive, achievement, self-development motivation), three types of extrinsic motivation for educational activities (self-esteem motivation, introjected, external) and amotivation.

Questionnaire "Diagnosis of Self-Organization Features" (DSOF) (A.D. Ishkov), which includes an integral scale "Level of Self-Organization" and six specific scales characterizing the development of the personal component of self-organization (willpower) and five functional components: goal-setting, situational analysis, planning, self-control, correction [5].

The study involved 199 undergraduate students studying at the Federal State Budgetary Educational Institution of Higher Education "VGSPU" in the fields of "Psychological and Pedagogical Education" (N=139) and "Psychology" (N=60) from 1st to 4th year of various forms of study (93.5% women). To form a comparison group with undergraduate students, only the AC questionnaire was proposed to master's students of the program "Personal Potential Development: Personalization and Digitalization" GAOU VO MGPU (N=25).

With the participation of deans' staff in the 2022-2023 academic year, a generalized expert assessment was given to part of the sample (N=97), identifying groups of successful and unsuccessful students. Unsuccessful students were those who regularly had academic debts, while successful students had consistently relatively high academic performance with no subject debts. Cases of unstable success were excluded from these groups, along with students for whom assessment was not performed for organizational reasons.

As a result of the first empirical study, the questionnaire was reduced to 69 items. Statements were excluded based on the analysis of the relationships of individual items with the success indicator, the consistency of the designed scales by Cronbach's alpha, and the "difficulty" of the items. It was also decided to abandon the detailed structure of 18 primary subscales and retain 6 scales as diagnostic indicators corresponding to the substantive model, and 3 – to the level model of academic competence.

Taking into account the feedback from participants who found it difficult to assess agreement with statements on a dichotomous scale, and to increase differentiation while reducing the number of items, it was decided to use a Likert scale with 4 levels of agreement and translate it into scores from 0 to 3.

Empirical study 2 was conducted in the spring of 2023 using a version of the questionnaire, including 69 items obtained after the screening in the previous stage. The study included the Student Educational Experience Questionnaire (SEEQ) (N.A. Lyz, E.V. Golubeva, O.N. Istratova), diagnosing indicators across 5 scales: satisfaction, intention to expand experience, self-efficacy and support, experience of self-regulated learning, engagement [7].

The survey involved 355 undergraduate and specialist students from VGSPU (N=283) and YKSUG named after Y. Kupala (N=72) aged 17 to 47 years (M=19.9; S=3.01), with first-year students making up 40.6%. Most of the sample consisted of students in teacher education profiles (N=200), also represented were: psychological and pedagogical education (N=98), special and defectological education (N=25), psychology (N=24), and other humanities profiles (N=8). The specifics of the university determined the predominance of women in the sample (90.1%). 91.3% of respondents are studying full-time.

Surveys in both studies were conducted using the Google Forms service, and participation was incentivized by incorporating the survey into work for psychological and pedagogical disciplines. At the same time, the preamble to the survey emphasized that participation was being considered, not the content of the responses. Attention was also drawn to the reflective nature of the survey and the opportunity to think about the education being received. According to the data from the groups where the author directly engaged students to participate, some of the least successful and engaged students ignored the invitation to complete the survey.

Statistical data analysis was performed using Jamovi 2.3 (CFA) and IBM SPSS Statistics 20.0 (other methods of analysis).

Results

To assess the conformity of the data obtained in study 2 with the structural-content and level models underlying the questionnaire, confirmatory factor analysis (CFA) was used. Calculations were performed twice, separately for each model. For the structural-content model, which assumes the identification of 6 substantive components within the AC, the fit indices were: CFI=0.620, TLI=0.606, RMSEA=0.0681. For the level model: CFI=0.613, TLI=0.601, RMSEA=0.0676. Despite the relatively low CFI and TLI values, the acceptable RMSEA values [17] allow both models to be considered at least not contradictory to the empirical data. Analysis of possible acceptable substantive adjustments to the models (excluding certain statements and changing the level to which the item was assigned) did not lead to a noticeable improvement in their quality, so it was decided to retain the a priori correspondence of items to the questionnaire scales. Factor loadings along with the content of the statements and processing keys are provided in the appendix.

Scale reliability based on consistency was evaluated by Cronbach's α. Values from the two studies are shown in Table 1. The distribution of raw scores for most individual subscales significantly deviates from normal according to the Kolmogorov-Smirnov criterion. However, the final score gives an acceptable fit to the normal distribution.

Table 1. Characteristics of the scales, indicators of consistency according to data from Study 1 (N=224) and Study 2 (N=355), significance of the deviation of the distribution from normal according to the Kolmogorov-Smirnov test based on data from Study 2

Notes: 1 Values of Cronbach's α coefficient for Studies 1 and 2 respectively; 2 Level of significance of the difference between the distribution and the normal distribution.

As with most scales of other self-assessment questionnaires included in the study (AMS and SEEQ), the distributions have a right-sided asymmetry with a shift towards high values (Table 2), which may be due to the insufficient coverage of the least engaged students. Therefore, standardization using sten or stena scales seems not quite adequate, but for rough estimation, it is possible to highlight average levels within the M±S interval. More justified, given the obtained data, seems to be standardization based on percentile values, which are presented in Table 2 along with the distribution parameters.

Table 2. Parameters of the distributions of values on the scales of the AC questionnaire according to study 2 (N=355)

Notes: 1 Decryption of the scale indexes is provided in Table 1; 2 M – arithmetic mean, S – standard deviation, As – skewness coefficient, Ex – kurtosis coefficient.

Table 3 presents the intercorrelations between the scales of the questionnaire. High and significant correlations between all scales indicate that the questionnaire as a whole diagnosis a fairly integral construct.

Table 3. Intercorrelations (Spearman's coefficients) between the scales of the AC questionnaire based on data from Study 2 (N=355)

Notes: 1 Decryption of the scale indexes is provided in Table 1; ** correlation is significant at p≤0.01 level.

To assess validity—both predictive and through the method of contrasting groups—comparisons were made (1) between undergraduate students with high and low academic performance based on expert assessments and (2) between a sample of undergraduate and master's students in the "Personal Potential Development" program using the Mann-Whitney U test. Predictably, the higher level of academic competence among master's students is due to their successful experience in undergraduate or specialist programs. The majority work in the education system and, at the time of the study, were handling academic tasks while managing high workloads, as well as engaging with the program's content focused on personal resource development.

The results obtained (Table 4) confirm that for most scales of the AC questionnaire, more successful undergraduate students compared to less successful ones, and master's students compared to undergraduates, show higher values at significant levels of difference.

Table 4. Comparison of contrasting groups by scales of the AC questionnaire based on data from Study 1

Notes: 1 Decryption of the scale indexes is provided in Table 1; 2 Significance level for differences tested by the Mann-Whitney U test.

Validity against external criteria was also checked through relationships with questionnaires on self-organization, academic motivation, and students' educational experiences (Table 5). Overall, the data obtained do not contradict theoretical expectations.

Table 5. Correlations (Spearman's coefficients) of the AС1 questionnaire scales with scales of the self-organization, academic motivation, and students' educational experience questionnaires

Notes: 1 Decryption of the scale indexes used in the column headings is provided in Table 1; * correlation is significant at p≤0.05 level; ** correlation is significant at p≤0.01 level.

Discussion and Conclusions

The validity of the proposed questionnaire was ensured through the development of the model, the involvement of specialists with extensive teaching and current student experience in the development and formulation of items, and their expert review. Empirical data confirming such types of validity as predictive and construct, based on the method of contrasting groups, and correlations consistent with theoretical expectations with data from previously standardized questionnaires on self-organization, academic motivation, and student experience, also supported validity. Reliability is confirmed by good indicators of internal consistency of the scales, as well as the acceptable consistency of the obtained data with the theoretical models underlying the development of the questionnaire.

High consistency of the questionnaire scales, although indicating that AC can be considered as an integral construct, somewhat reduces the value of the indicators obtained from individual scales. This can be considered a limitation when using the questionnaire solely for research purposes due to the relatively large number of items. However, when used within the framework of providing assistance to students in developing their AC, the volume of the questionnaire should not be an obstacle, as referring to different situations in which personal qualities and abilities manifest, resourceful for academic success, can already be useful.

The limitations of the proposed tool also include the insufficient representativeness of the sample in terms of coverage of different universities and fields of study, as well as gender. However, addressing the assessment and development of academic competence is especially important for students in psychology and education fields, as their own experience in improving the effectiveness of academic activities can be considered a contribution to the development of not only personal but also professional resources that support their future ability to help students solve similar problems.

Further directions for improving the proposed questionnaire include refining norms for students from other universities and fields of study, developing an express version of the questionnaire, and clarifying divergent validity by comparing with specific criteria for different scales. The question of comparing the capabilities of psychological diagnostics based on a self-report questionnaire and test procedures involving solving case tasks modeling problematic situations in learning remains open. An important direction for further research, we see, is the systematic identification and description using qualitative methods of the features of the manifestation of academic competence at the reproductive, reflexive, and functional levels in real academic activities, followed by comparison with the results of diagnostics using the questionnaire.

In the near future, the questionnaire may also need to be refined in connection with the changing specifics of higher education: the introduction of new forms of assessment (demonstration exam, etc.), the expansion of digital educational resources, including the use of tools based on artificial intelligence technologies, etc.

1. Shamionov R.M. et al. Akademicheskaya adaptatsiya studentov: razrabotka i validizatsiya novoi metodiki [Evaluating Academic Adaptation in Students: A New Technique]. Psikhologicheskaya nauka i obrazovanie = Psychological Science and Education, 2022. Vol. 27, no. 2, pp. 53–68. DOI:10.17759/pse.2022270205
2. Vodyakha S.A. Prediktory psikhologicheskogo blagopoluchiya studentov [The predictors of psychological well-being of modern students]. Pedagogicheskoe obrazovanie v Rossii [Teacher education in Russia], 2013, no. 1, pp. 70–74. (In Russ.).
3.  Gordeeva T.O., Sychev O.A., Osin E.N. Oprosnik “Shkaly akademicheskoi motivatsii” [“Academic motivation scales” questionnaire]. Psikhologicheskii zhurnal [Psychological Journal], 2014. 35, no. 4, pp. 96–107. (In Russ.).
4.  Grebtsova L.V., Timchenko S.V. Adaptatsiya studentov-pervokursnikov k obucheniyu v vuze kak odno iz napravlenii psikhologo-pedagogicheskoi raboty [First year students' adaptation to training at university as one of psychological pedagogical work directions]. Perspektivy nauki [Perspectives of science], 2011, 1(16), pp. 15–17. (In Russ.).
5.  Ishkov A.D. Uchebnaya deyatel'nost' studenta: psikhologicheskie faktory uspeshnosti [Student educational activity: psychological factors of success]. Moscow: Publ. AVS, 2004. 224 p. (In Russ.).
6. Leont'ev D.A. Samoregulyatsiya, resursy i lichnostnyi potentsial [Autoregulation, resources, and personality potential]. Sibirskii psikhologicheskii zhurnal [Siberian Psychological Journal], 2016, 62, pp. 18–37. DOI:10.17223/17267080/62/3 (In Russ.).
7.  Lyz' N.A., Golubeva E.V., Istratova O.N. Obrazovatel'nyi opyt studentov: kontseptualizatsiya i razrabotka instrumenta otsenki kachestva obrazovaniya [Students’ Educational Experience: The Conceptualization and Development of a Tool for the Assessment of Education Quality]. Voprosy obrazovaniya [Educational Studies (Moscow)], 2022, 3, pp. 67–98. DOI:10.17323/1814-9545-2022-3-67-98
8. Merkulova O.P. Bondareva L.V. Akademicheskaya kompetentnost' kak resurs professional'noi podgotovki psikhologa obrazovaniya [Academic competence as a resource for professional training of an educational psychologist]. In Nizhegorodtseva N.V. (Ed.). Teoreticheskie i prikladnye problemy psikhologii obrazovaniya: sbornik nauchnykh statei [Theoretical and applied problems of educational psychology: collection of scientific articles]. Yaroslavl': Publ. YaGPU, 2012, 14–19. (In Russ.).
9. Merkulova O.P. Praktikum akademicheskoi kompetentnosti. Kak uchit'sya v vuze uspeshno i samostoyatel'no? [Academic competence workshop. How to study at university successfully and independently?]. Saratov: Ai pi Er Media, 2018. 167 p. DOI:10.23682/72463 (In Russ.).
10.  Merkulova O.P. Akademicheskaya kompetentnost' kak lichnostnyi resurs uchebnoi deyatel'nosti studentov [Academic competence as a personality resource of students' learning activities]. Upravlenie kachestvom obrazovaniya: teoriya i praktika effektivnogo administrirovaniya [Education quality management: theory and practice effective administration], 2023, 4, pp. 29–38. (In Russ.).
11. Mikhailova N.S. Samoobrazovatel'naya deyatel'nost' i akademicheskaya kompetentnost' kak faktory akademicheskoi uspeshnosti studentov [Self-education activities and academic competence as factors of academic success of students]. Mezhdunarodnyi zhurnal eksperimental'nogo obrazovaniya [International journal of experimental education], 2016, no. 2, pp. 268–272. (In Russ.).
12.  Morozov A.I. Refleksivnaya pozitsiya v strukture akademicheskoi kompetentnosti studentov [Reflexive position in the structure of students academic competence]. Studencheskii elektronnyi zhurnal StRIZh [Student electronic journal StRIZh], 2020, 2(31), pp. 85–90. (In Russ.).
13. Nezhnov P.G., Kardanova E.Yu., El'konin B.D. Otsenka rezul'tatov shkol'nogo obrazovaniya: strukturnyi podkhod [Assessing school outcomes: a structural approach]. Voprosy obrazovaniya [Educational Studies (Moscow)], 2011, 1, pp. 26–44. DOI:10.17323/1814-9545-2011-1-26-43 (In Russ.).
14. Smirnov S.D. et al. O svyazi intellektual'nykh i lichnostnykh kharakteristik studentov s uspeshnost'yu ikh obucheniya [About the connection between intellectual abilities and personality traitsof students and their academic success]. Vestnik Moskovskogo universiteta. Seriya 14: Psikhologiya [Lomonosov Psychology Journal], 2007, no. 3, pp. 82–87.
15. Privalova S.Yu. Samoregulyatsiya uchebnoi deyatel'nosti i lichnoe razvitie studenta [Self-regulation of educational activities and student’s personal development]. Izvestiya Saratovskogo universiteta. Novaya seriya. Seriya: Sotsiologiya. Politologiya [Izvestiya of Saratov University. Sociology. Politology], 2011. Vol. 11, no. 1, pp. 27–30. DOI:10.18500/1818-9601-2011-11-1-27-30 (In Russ.).
16. Asmolov A.G. et al. Proektirovanie universal'nykh uchebnykh deistvii v starshei shkole [Designing universal learning activities in secondary education]. Natsional'nyi psikhologicheskii zhurnal [National Psychological Journal], 2011, no. 1(5), pp. 104–110. (In Russ.).
17. Sorokova M.G. (ed). Rukovodstvo po standartizatsii psikhodiagnosticheskogo instrumentariya: trebovaniya i otsenka kachestva [Guidelines for standardization of psychodiagnostic tools: requirements and quality assessment]. M.G. Sorokova, E.Yu. Kardanova, N.P. Radchikova, V.V. Fedorov. Moscow: FGBOU VO MGPPU, 2024. 48 p. (In Russ.).
18. Yukhno V.S., Denisova E.G. Affektivno-kognitivnye prediktory akademicheskoi uspeshnosti sovremennykh studentov: analiticheskii obzor [Affective and cognitive predictors of academic performance of contemporary students: an analytical review]. Severo-kavkazskii psikhologicheskii vestnik [North-Caucasian Psychological Bulletin], 2022. Vol. 20, no. 4, pp. 17–26. DOI:10.21702/ncpb.2022.4.2 (In Russ.).
19. Owens J.S. et al. Evaluating a Short Form of the Academic Competence Evaluation Scales: Expanded Examination of Psychometric Properties. School Mental Health, 2020. Vol. 12, no. 1, pp. 38–52. DOI:10.1007/s12310-019-09347-9
20. Kosykh A.V. Features of adaptation of 1st year students to study at a university. Nauchnye gorizonty [Scientific horizons], 2020, no. 9(37), pp. 43–51.