Measuring Higher-Order Cognitive Skills in Collective Interactions with Computer Game

The present study is focused on testing the computer game system ‘PL-modified’ as a diagnostic tool for measurement of higher-order cognitive skills by middle-school students in individual and collaborative game. The ‘PL-modified’ is a computer system designed as a game which implies a set of concrete parameters specially elaborated for assessment of the cognitive actions of analysis, planning, and reflection — the basic higher-order cognitive functions which determine high achievements in school education according to the Russian theory of developmental education. 189 middle-school students at the age of 11—12 years participated in this study. Two research questions were asked: 1) whether the cognitive actions of analysis, planning, and reflection measured by special markers of the computer game system performance are correlated with each other as a valid indicator for the new constructed diagnostic instrument; 2) which type of the game — individual or collaborative — provides better conditions for manifestation of the above mentioned higher mental actions. Abstract intelligence as an additional anticipated factor for high game performance was also assessed and controlled. It was revealed that participants exhibit the higher level of the cognitive actions of analysis and planning in collaborative game. At the same time the patterns of the interactions between the researched variables as well as distinct parameters of game performance are determined by the concrete level of intelligence which rather varies in different pairs of collaborators. We discuss our results from the position of the further prospects for the application of the ‘Pl-modified’ computer system as a potential instrument of measurement and development of higher-order thinking actions. In terms of the modern educational programs teachers need simple diagnostic tools for measurement of school-children’s thinking development. Traditional ‘pen-and-paper’ techniques become quickly outdated as much as they may not be sufficiently motivationally attractive for children and focus only on the result of the thinking process. In this regard, such diagnostic instrument formed in the format of a computer game and centered on the whole gaming process allows fixing children’s actions and provides important information on the dynamic characteristics of their mental process.

 

1.    Introduction

The present study aims at testing the higher-order cognitive skills level measured by the ‘PL-modified’ computer game system in individual and collaborative game. The relevance of the present research task is determined by the practical requirements formed for the diagnostics nowadays. The growing usage of computer technologies, various devices and the Internet in different social domains also outline high challenges for the diagnostics criteria in education. Therefore, in its turn, the elaboration and the usage of a valid and accurate diagnostic tool to measure cognitive skills more precisely is getting a big importance.

On one hand, diagnosticians possess many of standardized psychometric tests whilst conducting studies. On the other, numerous ‘pen-and-paper’ tests are rather outdated so far they do not always correspond to the overriding provisions of the diagnostic conditions. This is more alarming for school-children who are often less motivated while being tested and feel worry for the whole diagnostic situation which can lead to incorrect data. Moreover, the social situation of growing up of kids today triggers the problem of the ecological validity of the present psychometric tests which also do not simply reflect real activities of the new generation of schoolchildren. Taken the above mentioned into account, the modern tendency to explore potential of diagnostic techniques by using various games or game components so far children are rather fascinated with computer games nowadays (Voiskunsky & Bogacheva, 2017) is getting to be more pronounced.

Gamification effect research has already affected a variety of different domains over the last decade (Fo-roughi et al., 2016; Gallagher & Grimm, 2018; Margo­lis et al., 2018, 2020; Quiroga et al., 2015, 2016). Many researchers stick here to the ‘Digital Game-based Science Assessment’ approach by incorporating the game context itself into the educational process that brings numbers of advantages from the diagnostic point of view (Chu & Chiang, 2018; Godwin et al., 2015; McClarty et al., 2012). First, a game situation increases attractiveness to schoolchildren and has general positive motivational effect (Lumsden et al., 2016a). Another apparent benefit from gamification implies to bring specific various characteristics of computer games such as competitiveness, player communication skills, and active use of visualization techniques into appropriate diagnostic tool. It means that these cognitive techniques structured in a game format capture children’s mental activity in its dynamic that corresponds to real life situations. The present points draw in general rather realistic possibilities in application of computer games as an up-and-coming psychometric instrument in diagnostic process since such ‘gamification’ tools allow covering not an isolated action or a concrete trait, but rather complexes of various interrelated psychological qualities including social skills and meta-cognitive knowledge that are definitely required to be successful in learning process and then in modern world.

To sum up, the research question about the possibility to use a controlled computer game for the assessment of certain mental skills is considered as being reasonable and worth of further examining. Based on the popularity of some modern computer games and classification of game types (achievement/challenge-, immersion-, and social-based games by Hamari & Tuunanen, 2014; Koivisto & Hamari, 2019), we elaborated the special computer-like game system entitled ‘PLmodified’ to analyze the higher-order cognitive actions’ level by middle-school students. The system rests on the principles of popular game ‘Lines’ where randomly appearing balls are needed to be somehow organized on a game field. So the present game is quite easy to be learned and brings a touch of entertaining. At the same time a number of distinct differences were set up for each game step. Thus, the whole game challenges participants to use their mental skills for getting more points. Another one challenge brings a collaborative game when children are divided in pairs and stimulated to collaborate and communicate with each other to be advancing playing the game. Such circumstances should aim to consider which research possibilities trigger the realization of concrete cognitive actions in this game format when two partners are needed to constantly collaborate to be successful. In this case the main effects on game performance compared in two types of the game will be significantly under the spotlight of this study.

2.    Theoretical background: higher mental
actions and the theory of developmental
education

The methodological basis of the present study grounds on the cultural-historical activity theory (Vygotsky, 1935; Leontyev, 1959) and the theory of developing education (Elkonin, 1966; Davydov, 1996) postulating that human’s mental development is determined by the attribution of generalized modes of actions in communication. Therefore, collaborative activity realized through communication promotes an active position of collaborators themselves and triggers development of their mental actions being at the core of theoretical thinking which is considered to make essential impact in educational process from the very beginning. Davydov (1996) allocated three key components of theoretical thinking process highlighting the mental functions of the higher order (or the higher mental actions) such as analysis, planning, and reflection. Formal theoretical level of analysis (in contrast to empirical) is aimed at highlighting the internal, essential features in the studied phenomenon which allows attributing an object to a specific class. The cognitive action of planning as a part of the more general ability to act ‘in mind’ is interpreted as the ability to predict what will happen to an object if certain transformations are made (Brown & Bransford, 1982). Reflection as the ability to see the origins of one’s own way of action represents the skill to distinguish a universal relation in a studied object.

Taken together, these three so called learning actions considerably contribute to the high attainments in education by setting up the ground to understand main concepts of school disciplines and to widely hone reasoning skills. Moreover, collaboration and any other forms of children’s cooperative activity seem to play a pivotal role in the mental actions’ development. The advancement of those preschool- and school-students who are skilled well enough to interact in small groups and cooperative games in their conceptual thinking and better learning performance were shown in number of contemporary studies (Guruzhapov, 2000; Gordeeva, 2019; Davydov, 1996; Roubtsov, 1996; Polivanova, Rivina, Ulanovskaya, 2013; Tsukerman, 2016, 2020). All of them stress that namely collaborative activity allowing children to mentally hold equal points of views together paves the way of how the conceptual thinking will be finally formed. Collaborative problem solving is also assessed by PISA (2017) as the key abilities to communicate, manage a conflict, and organize a team to be successful in many social domains.

Taking these empirical studies into account we presume that introduction of a computer game into children’s collaborative work brings an important input in the understanding of psychological aspects their interaction and — above all — the manifestation of their cognitive actions. In Russian psychology there are a lot of studies focused on different points of collaborative activities using a computer (Kritsky, 1989; Kuravsky, Baranov, 2005). For example, Kritsky and Shcherbinin (2007) organized of computer-mediated communications in educational activities and demonstrated how much the division of individual operations within a necessity to perform a certain action with the object of study together can promote collaboration and communication between students by fostering their ability to act jointly in a way which is corresponding to the principle of the studied object. Specifically, control over the communication process (in particular, the consequent narrowing the channel and modes of communication in messages) can contribute to change of the nature of communication and discover a sign-symbolic (mathematical) form of representation of the studied object. The analysis of partner message texts provides valuable information about the level of interaction and the target orientation of the subjects’ communication.

Which conditions are required to organize collaborative learning process promoting to acquire the generalized mode of actions? A division of individual operations under conditions of necessity to perform jointly seems to be a right decision for that because it suspends the whole activity by stimulating to understand additional characteristics of the activity itself and to coordinate own actions within joint context and express arguments in a favor of one’s own action and a partner’s action as well. In this study the ‘Pl-modified’ computer system creates such type of the game collaborative problem­solving when partners cannot further play unless they are working on each step together and discussing every move they take. So, the communication process delivers here not conditions, but first of all means of collaborative actions. Participants encountered with the necessity to analyze and reflect their and joint moves to create general mode of actions.

Considering the previews facts we formulated the main hypothesis of the study stating that participants will exhibit the higher level of the cognitive actions of analysis, planning, and reflection in collaborative game in contrast to individual game. These differences will be specified in the faster dynamic of getting the game points, the higher total game points, the better comprehension of the game rules, and its changes in different cycles.

3.    Material and methods

1.   Participants

189 middle-school Russian speaking students (46% of girls and 54% of boys, age range: 11—12 years) participated in our study. The whole sample included participants from 5th and 6th classes of two traditional city schools.

2.    Materials

3.2.1    The ‘PL-modified’ computer game system as a diagnostic instrument

The whole computer system represents the game field with the design of 9 x 9 cells where the balls of different colors appeared. Three balls were displayed at each game turn. Participants had to move one ball by one mouse­click in a free space of the field in a way to build a vertical, horizontal or diagonal line of five or more balls with the same color. After such line of colored balls has been built, it completely disappeared and participant earned his points. More points were given for the longer lines consisting of six and more balls.

One important remark is here worth of notion. Unlike the standard version of this game elaborated by Gamos Company where the balls are displayed randomly, the present research version implies certain sequences due to a principle which might be changed in different cycles of
the game in a hidden way by stimulating students’ reasoning about the possible regularity of how the balls are appearing each time. We assume that these sequences or so-called ‘rules’ of the balls’ (and students’ attempt to discover and use it while planning their moves) will booster the whole game process by raising the chances of students to earn more points. Thus, three rules for each game set were elaborated. For example, for the first game set the following algorithm was prepared: 1) Each row of the game field implies one distinct color of the balls which is repeated at regular intervals of two succeeding lines (the sequence of red — blue — green lines of balls every time); 2) each next ball of a distinct color succeeds in the next through one vertical cell (it means that if the first red ball appeared in the ‘a1’ cell then the next red one will be displayed in the ‘a3’ cell meanwhile the third one will be at the ‘a5’ cell position); 3) the whole game field is divided into two sections implying the ‘sequent balls’ section and the ‘random balls’ section. In the ‘sequent’ section balls were displayed in accordance with the above mentioned rules whereas there was no special logic for the balls’ presentation in the ‘random’ section — that is how this section served as kind of a ‘storage’ for the balls of different colors which could be used for the lines construction. To sum up, there were three balls appeared in a different way every time. The first one was of a distinct color and was displayed at the special cell position in conformity with the first two rules in the ‘sequent’ section. Another two balls appeared in the ‘random’ section, in so doing, always to be of different colors.

The described principles of the balls’ presentation were carefully created for each game set by taking the age of participants into account. So, they involve certain flexibility whilst keeping on changes for a special need to make them more or less complicated. The examples of both game field and balls construction are displayed in Figures 1 & 2.

Transformation of original computer game in research environment was carried out using the following programs: HTML, JavaScript, jQuery library, Ajax technology and MySQL DBMS.

3.2.2    Research procedure

Participants played two games set in two different conditions: first individually and then together with a partner. Before the main procedure started participants were exposed to two minutes short test version to make their first acquaintance with the game process. The whole procedure was going online by clicking at a special link. Each of the two games consisted of three sets. Thus, participants were dealing with three different algorithms playing each game. Every algorithm was lasting for 10 minutes. And each game was lasting for 40 minutes (one lesson), respectively.

First individual game started when every student was working separately by sitting in front of a personal computer. At the second lesson online collaborative game started. Participants were randomly divided in pairs in advance. In these game conditions students have also build up the lines of the balls, but they must have to coordinate their individual actions at each joint game step. It means that every time when one partner makes a move, it is popping up on the game field of the second partner who is able to ‘approve’ or ‘stop’ this move. In case of the ‘stop’ option the first partner had to change his decision by making another one move whereas in case of the ‘submit’ option it is the second partner’s turn to play. The example of such game field of collaborative game problem solving is displayed in Figure 3.

Participants were given an instruction whereby they were informed about the new game and intentionally paid intention to collaborative type of the task. Therefore, they have to interact with each other discussing their possible and actual game actions. All dialogues were recorded by google_meet online system and voice-recorder program.

Three variables represented three cognitive actions were assessed: 1) the analysis defined as the ability to discover the rule which determine sequences of the balls appearance; 2) the planning defined as the ability to use the very knowledge about the rules and to use it whilst playing the game by predicting appearance of a certain ball in a certain place; 3) the reflection defined as the ability to understand change of the previous rule in a new one in each succeeding game set and to restructure own actions in accordance with new rules. We used the following scoring strategies to measure these cognitive skills by certain methods. So far that the cognitive action of ‘analysis’ in its clearest meaning implies the ability to discover and understand the rule the following scoring strategy was introduced. A special list with descriptions of several rules after each game set was created. Some of such descriptions correctly characterized the very principles that were just put into the concrete game set. Each list included nine descriptions with three correct and six wrong rules. Participants were asked to pick amid all descriptions those which in their eyes precisely match the sequences that they faced during a game set with. The number of the descriptions picked correctly after each game set and that in total were scored and used as an ‘analysis’ variable — in this case understood as awareness of the rule. In the collaborative game the number of picked descriptions was scored for each partner separately As for the cognitive action of ‘planning’ the number of play points for an each game set and in total was calculated. In the collaborative game mode the total number of points was scored as an overall contribution to the game performance, and was attributed to each participant of a pair.

The cognitive action of ‘reflection’ was the most difficult variable for our measuring process. In general, this cognitive action involves a deep thinking process of understanding that something goes wrong as a starting point, the ability to recognize the beginning of his/ her way of action, to redefine his own actions and to set them in a new direction. In terms of the play process the ability of a child to be reflective implies his comprehension that the way of action which was effective in respect to the rule in the preceding game set does not work in the new one, and he needs to find out this rule and construct a new strategy again considering ongoing balls sequences. Thus, it seems to be obvious that more ‘reflective’ gamers will not chaotically move the balls, but will structure lines of the balls of the same colors on one side of the field and wait for an every further move to see which new balls will arrive. In other words, they will interrupt in a way their own play trying to collect more data for construction of a new strategy by moving from
play to search and explore exploration. Previously made pilot studies let us be sure that such gamers will have the very structured game field at the end of the game. So, finally we used total amount of free cells on the last game step as the most precise variable for the variable ‘reflection’ in the further statistical analysis.

The whole game process and the main researched variables are displayed in Figure 4.

At the third lesson abstract intelligence was measured by the Raven’s Advanced Progressive Matrices by J. Raven (2002). The test consists of twelve tasks with increasing difficulty. Each task is presented in black ink on a white background and represents sequences of abstract figures in horizontal and vertical lines in 9 x 9 design. The last ninth figure is absent. Participants have to complete the line by finding this ninth figure in accordance with the logic of presented lines of figures. The whole testing procedure was lasting for fifteen minutes. Students earned one point for each correct task. Therefore the total test productivity implied the number of correct answers.

4.    Results

4.1    The whole sample: main effects

Mean scores were calculated for the three variables representing intelligence, and two cognitive actions of ‘analysis’ and ‘reflection’, respectively. As for the cognitive action ‘planning’ special calculations were additionally done. It is clear that participants are in unequal conditions playing individually or with a partner, having more time to coordinate their moves together, and getting less point from the beginning. Thus, the qualitative game performance parameter was calculated in that way:

X-parameter = X1/X2,

where X is the game performance (or ‘planning’) itself, X1 is the total points in each game set, and X2 is the number of the moves in each game set.

All means were counted and compared for all variables in two game conditions. A Wilcoxon Test was made on the comparison of the means. Results are presented in Table 1.

The data included in Table 1 show advantages in means by two of three variables in collaborative game. These are the cognitive actions ‘planning’ and ‘reflection’. Additionally we analyzed means of the X-parameter for each game set. These results are depicted in Figure 5 and demonstrate different patterns of game performance from the first up to the last game set by decreasing in individual game and increasing in collaborative conditions. Thus, participants exhibit better game performance playing in collaboration.

At the next step the correlation analysis between all researched variables was done. The coefficients matrix is presented in Table 2.

Table 1

The main differences in means by comparison of two game conditions (SD are displayed in parentheses)

Measure	Individual game	Collaborative game
‘Analysis’ (No. of correct rules)	3,33 (2)	2,56 (1,61)*
‘Planning’ (X-parameter of game performance)	4,1 (1,22)	6,44 (3,38)*
‘Reflection’ (free cells on the last game step)	72,24 (39,38)	121,57 (28,24)*

differences are significant at the P-0.000 level.

The results presented in the Table let us see two key effects. First, the ‘analysis’ parameters significantly correlate with each other in two types of the game. And second, there are essential interactions between researched variables in each game. In individual game variable ‘planning’ correlate with both variables ‘analysis’ and ‘reflection’ whereas in collaborative game there is only one interaction between ‘analysis’ and ‘planning’. As for the variable ‘reflection’ it doesn’t show any correlations with ‘analysis’ in any type of the game. Such effect seems to be interpreted taking the game type into account. In case of collaborative game participants have free cells in the end anyway so far they spend much more time to make each move. Thus, in such conditions their game field is more ‘clear’. But this fact can be caused by the game quality or low motivation when they
waste a time instead to work meaningfully. Therefore, in collaborative game cognitive action ‘reflection’ shifts from individual’s thinking process and goes up to the level of discussion and communication. We took another scoring procedure for ‘reflection’ in further data analysis.

4.2    The data analysis in groups divided by the intelligence criterion

Next step concerns the data analysis in groups. As we previously mentioned participants were divided in pairs by chance. It means that pairs (as well as two partners in one pair) could widely differ in their cognitive abilities Based on this fact all couples were classified by the intelligence criterion. For this aim the result of APM-test was marked as the ‘high level’ (above 66,7% of the whole sample), the ‘middle level’ (lies between 66,7% and 33,3%), and the ‘low level’ (less than 33,3%) for every student. After that two main groups were revealed, namely: the group 1 with two partners (58 students in total) of the equal level of intelligence, and the group 2 with two partners (70 students in total) of the different levels of intelligence. Some of the data (10% of the sample) were excluded from the current analysis because of their defect or very low intellectual performance.

At the first step the comparative analysis in mean values for all researched variables was accomplished. The results are displayed in Table 3. This Table doesn’t include means in reflection because it was previously clear that the scoring procedure used for the data in individual game doesn’t match for the data in collaborative game. Therefore, such means seem to be impossible to compare, and we focus on ‘analysis’ and ‘planning’ mental actions more precisely.

Table 2

The correlation matrix of the interactions between researched variables in two types of the game

Variable	1.	2.	3.	4.	5.	6.
1. Analysis (individual)
2.Analysis (collaborative)	0,35**
3.Planning (individual)	0,22**	0,20*
4.Planning (collaborative)	0,2*	0,21*	0,17
5.Reflection (individual)	-0,14	0,04	0,48**	0,06
6.Reflection (collaborative)	-0,03	0,02	-0,03	-0,1	0,06

*p = 0.05; **p = 0.01.

Table 3

The main differences in means by comparison of two game conditions in two groups (SD are displayed in parentheses)

Measure	Group 1		Group 2
Game type	Individual	Collaborative	Individual	Collaborative
‘Analysis’ (No. of correct rules)	2,63 (1,96)	2,44 (1,56)	4,02** (1,8)	2,75 (1,63)
‘Planning’ (X-parameter of game performance)	4,15 (0,99)	5,9 (3,51)	4,43 (1,24)	6,89* (3,04)

**p = 0.02; **p = 0.001

 

ANOVA analysis was used to compare all means. First, it was revealed that participants in the group 2 outperform those from the group 1 in both ‘analysis’ and ‘planning’ in individual game (F = 13,18; р = 0.001; F = 2,79; р = 0.02). Second, individual game conditions provide better chances for ‘analysis’ in comparison to
collaborative game. No matter which group is taken for consideration. But completely reversed effect is revealed for ‘planning’: in both groups participants are better in getting the points in collaborative game. Moreover, students in the group 2 demonstrate the higher level of means in ‘planning’ than students from the group 1. Significantly, correlations between intelligence and any means in game performance were not shown.

At the second step patterns of the interactions between researched variables were analyzed on each game set. The main effects are shown in Figures 6 &7, and demonstrate different effects on the correlations. In the group 1 the connections between two researched parameters of mental actions is rapidly growing from the first till the last game set. In the contrary, for the group 2 a small acceleration of the correlation coefficients is clearly seen only for individual game, but not collaborative.

To sum up, the present results lead to the following conclusions. Participants exhibit higher game performance in collaborative game. Such effects are shown in total game points as well as on each game set. Furthermore, different intellectual sources provide additional significant conditions. Students with an equal level of intelligence (the group 1) demonstrate better connection between two higher-order cognitive actions of analysis and planning through the whole game process.

4.3    Partners’ dialogues analysis as cognitive action reflection’ in collaborative game

All participants’ dialogues were transformed into the written texts after being listened auditory. The text was prepared for every student and for each game set. All expressions were written without being skipped for a detailed analysis. After that the phrases were divided into six categories by the criterion of participant’s attitude to the game and to his / her partner. In the end each phrase was awarded by one point for a concrete category. The categories and expressions examples are presented in Table 4.

Not all categories are meaningful as the indicators of the ‘reflection’ variable. For example, the first and the
second don’t have a significant impact on the thinking unless they change game behavior trajectory. But such expressions provide emotional attitude and catalyze generally the game. The third and the fourth category are of a big interest because of their focus on the participant himself or his partner. In this case the fourth ‘collaborative’ intended category should be correlated with the ‘planning’ variable as a key parameter of game performance. Finally, the fifth and the sixth categories play the pivotal role by having a substantive direction to the whole game situation. The phrases from these categories are signified as generalized markers of the ‘reflection’ insofar as they concern to expressions about concrete rules and reflect arguments of how to act to be successful in the game.

Means for the whole sample and two groups are displayed in Table 5.

From the present Table we can derive significant differences between two groups of participants almost in each category excluding the fifth. All means are more frequent in the group 1 and show that students with an equal level of intelligence tend to more often interact with each other using different expressions including as much neutral or emotional as meaningful.

At the next step the correlation analysis between the researched parameters — variables ‘analysis’, ‘planning’, and each category displays the variable ‘reflection’ — was accomplished. The main results are presented in Table 6.

Table 4

The list of the categories with concrete phrases examples

Category	Expressions
1. Neutral (doesn’t change behavior of a participant)	«Shall we begin», «What’s that? », «I can’t», «Have you moved in this way?», «Where are you moving!»
2. Neutral-motivational (it doesn’t participant’s behavior in general, but it brings emotional / motivational investment in the dynamic of the game)	«Common, move on! », «Please, approve», «Hurry up, we’re just running out of time», «Common! », «Ah! All right! », «Yes, let’s make it in this way»
3. Individually intended (when partner talks about his actions or asks his partner to pay attention to his actions)	«I’m making vertical line», «I’m making horizontal line», «I’m making horizontal line», «Amid / cancel my move».
4. Collaborative intended (when participant talks to his partner about his moves or their collaborative actions and plans)	«Make it again», «Let build this line together», «Better to take this ball», «Take balls of the other color» («take balls from this angle... green...blue» etc.), «We need to clear the field», «We need to try this».
5. Agreeing (participant agrees with his partner and accepts his move or cancels his own move, but understands the reason)	«Generally, yes, it is so», «Yes, we take the green one», «I accept», «Yes, I agree, we move in this way», «Yes, all right».
6. Changing (objecting) (an attempt to change partner’s behavior with concrete arguments or statement about the game rules)	«We can’t build a line in this way», «This move is useful», «This move doesn’t bring anything», «This move will destroy a line», «Diagonals are coming this way», «We could get more points in such manner», «It’s easier», «Three blue balls are arriving one after an- other» etc.

Table 5

Means of all categories

Category	Whole sample	Group 1	Group 2
1. Neutral	3,61	4,63	2,86**
2. Neutral-motivational	16,20	21,1	12,58**
3. Individually intended	3,91	4,1	3,79*
4. Collaborative intended	13,44	15,00	12,28**
5. Agreeing	1,14	1,13	1,16
6. Changing (objecting)	1,72	2,19	1,37*

*p = 0.05; **p = 0.01.

The present correlations demonstrate general patterns of the interactions no matter which sample is under the spotlight: the robust relation between the ‘planning’ parameters and means of the last three categories (i.e. ‘collaborative intended’, ‘agreeing’, ‘changing’) is significant for the whole sample and also for each group.

Table 2

The correlation matrix of the interactions between researched variables in collaborative game

	Whole sample		Group 1		Group 2
	Analysis	Planning	Analysis	Planning	Analysis	Planning
1. Neutral	-0,23*	-0,37**	-0,28	-0,51**	-0,13	-0,22
2. Neutral-motivational	-0,34*	0,11	-0,24	0,22	-0,32*	0,34*
3. Individually intended	-0,13	0,44**	-0,25	0,18	-0,1	0,61**
4. Collaborative intended	-0,15	0,60**	0,1	0,80**	-0,27	0,63**
5. Agreeing	0,02	0,51**	0,27*	0,82**	-0,17	0,39*
6. Changing (objecting)	-0,18	0,49**	-0,1	0,72**	-0,22	0,55**

*p = 0.05; **p = 0.01.

Thus, we can conclude the deep connection between such variables as ‘planning’ and ‘reflection’ which represent two key mental actions for successful problem solving.

Another significant interaction ties ‘planning’ parameter with the mean of the fourth, ‘collaborative intended’ category. This empirical fact let us discuss about the validity of participants’ dialogues as significant markers of their verbal reactions catalyzing the game process. The effect of the stronger correlation between ‘planning’ and means of the three last categories popping up in the group 1 is also important of additional notion. It points to better mental possibilities for successful game problem solving.

On the other hand, weaker correlations are obvious when it comes to the variable ‘analysis’: its connection to the categories whether negative or of low significance. The exception concerns again the group 1 where positive correlations are seen in case of the fifth category. Thus, this is the group 1 which can be an appropriate example of the interactions between three researched higher mental actions of analysis, planning, and reflection.

5.    General discussion

The present study was aimed at testing the computer game system ‘PL-modified’ as a diagnostic tool for measurement of higher-order cognitive skills by middle-school students in individual and collaborative game. For this purpose two versions of the system were elaborated: the first one is for an individual game, and the second one was set for a collaborative game where participants had to work together — at the verbal and behavioral level by splitting individual game actions under conditions of making a joint game decision (step) — to get more points. All parameters of game performance were compared then for the whole sample as well as in two groups divided by the intelligence level criterion.

The results let us derive the following key conclusions. First of all, the main parameters of game performance assessed by the proportion of total game points to the number of game moves are higher in collaborative game. These effects are to be increasing at each game set by keeping positive game dynamic. Such results partially confirm the hypothesis that participants will exhibit the higher level of cognitive actions in collaborative game in contrast to individual game. These differences are specified in the faster dynamic of getting the game points, and the higher total game points. Thus, we can also deduce that this is the collaborative game which provides good conditions for participants learning to communicate and better understand each other. Unlike the individual game where game performance decrease can be caused by faster fatigue and exhaustion of motivation to further play.

At the same time, the above mentioned effects are relevant for the cognitive action planning, but not for the analysis which parameters were higher in individual game. We assume that such results can be partially explained by the very way of how the variable was assessed. In all cases participants picked the rules individually, even in collaborative game. This measurement procedure was initially introduced because of some technical reasons. We also aimed to separate the input of every kid in rules awareness and to stick to more valid conditions by making the data statistically comparable. Nevertheless, the correlation analysis revealed significant interrelations between the data of the ‘analysis’ variable in two types of the game that confirms the correctness of the used measure technique. Thus, we can conclude that participants simply see more rules in individual game.

On the other hand, the correlations between the variables of ‘analysis’ and ‘planning’ demonstrated more robust interactions in collaborative game — the fact that stresses deeper connections of two mental actions in collaboration. As for reflection then this variable was the most difficult to be scored because of qualitatively various game conditions. Thus, in case of collaborative game the micro-analysis of the dialogues was undertaken and revealed significantly high correlations between parameters of ‘planning’ and ‘reflection’, and (in some cases) ‘reflection’ and ‘analysis’. In general, the interactions between three researched higher cognitive actions obviously manifest themselves in collaborative game. The present empirical facts statistically confirm that three researched variables are correctly represented and scored by the elaborated game procedure. Moreover, these main data confirm those revealed in the previews study (Margolis et al., 2020) about the relations of the mental actions of analysis, planning, and reflection. For its part these data concern to our first research question about the correlations of the cognitive actions as a valid indicator for the ‘PL-modified’ computer system and make to outline appropriate prospects for its usage as a potential diagnostic tool in education.

Another one important result highlights the impact of individual differences on the major effects. Both types of game performance seem to be determined by the level of the higher-order cognitive actions of analysis, planning, and reflection as well as the patterns of their interactions. However, manifestation of the above mentioned cognitive actions is additionally caused by the factor of different cognitive abilities (for this game there is equal / unequal intelligence) as well as main game results depend (qualitatively and quantitatively) on cognitive abilities of partners’ interactions. The pairs with an equal intellectual level generally play more effectively. This impact is shown in positive dynamic of the interactions between game parameters and (in some cases) higher final game performance.

Such results coincide to a certain extent with those detected by Perret-Clermont(1991) who stated that cognitive development is determined by a special form of human’s interaction when a person deals with various thoughts and meanings. A child stimulates his own development by coordinating his opinion with others, and summoning socio-cognitive conflict which in its turn leads to intelligence development. Taking this position into account we were not focusing on the intellectual development precisely, but considered children cognitive sources with a big attention. And it turned out that namely this factor — of an equal intellectual level — played an important part in general effects.

Finally, the meaningful and thoughtful collaboration of partners provides better comprehension of the game rules, and as a result better game itself. This outcome is expressed in higher game points. Thus, the real intellectual activity in collaborative work seems to be more difficult than the traditional scheme of a direct connection between an action and an output. And forms of the partnership create space of possibilities which encourage potential patterns of interactions under different psychological factors.

To sum up, the present conclusions let consider the key collaborative game parameters as additional diagnostic markers for the level of higher mental actions by middle-school students so far the effects were shown on the whole sample as well as in two divided groups. Meanwhile this is worth of notion that there are not direct and simple connections between assessed variables without consideration of various internal and external factors. Collaborative game doesn’t simultaneously lead to a better result, but does form many start-up chances for potentials expressed in concrete forms of mental actions.

For the computer game system itself the results let clearly characterize the potential advantages for its usage as a supplemental diagnostic tool since the system obtains psychological and technical possibilities including time limits, big motivational attractiveness, its speed, easiness and also accuracy whilst empirical data will be testing and processing. Furthermore, the system implies measurement of multiple cognitive actions at one time period and in terms of one type of activity which involves not just simple gaming process, but also information analysis and thinking of a strategy before trying to solve a problem. In this context the present computer system can be also observed as a learning tool which is supposed to enhance players’ problem solving skills. Some positive steps were done by Adachi & Willoughby (2013) in this domain. They argued that exactly strategic video-games promoted self-reported problem solving skills and indirectly predicted academic grades. We gather that the current findings let the present system to be incorporated into educational process in the future. The measurement of the whole process of the present problem solving while playing a game let to analyze more comprehensive presentation of cognitive actions in their internal form (as mental actions). On the other hand, such representation appears also as a form of social interactions by creating not only a context of joint playing actions, but manifesting itself in external way by a dialogue with a partner of communication. In general, these modern game-like diagnostic tools allow studying more subtle psychological phenomena and overcoming traditional static measurements that capture only single elements of a researched object instead of considering their real interactions. Thus, the elaborated ‘PL- modified’ computer game system seems to be more valid as a new potential instrument taking its closer correspondence with the real thinking process during a concrete (here game) activity into account.

5.1     Limitations of the study and its further prospects

While the main hypothesis of the study was confirmed, some limitations should be acknowledged. First, we studied and compared the effects on the sample which includes only two schools. The fact that school children from one school can have some cognitive edges and advantages in learning process must be taken into account. Although the intellectual impact was deliberately controlled and wasn’t strongly revealed on the whole sample which means that such participants couldn’t have and use any intellectual benefits from game performance in advance. Secondly, the potential gender differences were not analyzed. Since female participants are generally more motivated by challenge games (McDaniel et al., 2012), some gender effects should be taken in the focus of the future studies.

In general, the whole diagnostic potential of the computer system must be anticipated with a sufficient caution since it is not fully studied. The following research prospects are worth of being outlined:

1.   the researched sample must be extended by recruiting students of different ages, academic performances and those who stick to diverse educational processes and learning methods. Such methods will allow expanding the psychological relevance of the results up to the general sample of Russian school-children;

2.   an increasing number of cognitive tested parameters makes the accurate and differential measure of students’ cognitive potential more possible;

3.   the flexible properties of the whole game system imply varied forms of the organization of the game process in the future. As an example in certain circumstances the collaboration of participants whilst they are solving game tasks not exactly in pairs, but in small groups will allow to study their different individual traits including social intelligence, communicative abilities, and other social and cognitive patterns.

6.    Conclusions

The present study was aimed at testing the computer game system ‘PL-modified’ as a special diagnostic tool for the measurement of higher mental actions by middle-school students in individual and collaborative game. We took the sample of children at the age of 11—12 years to prove whether the higher mental actions of analysis, planning, and reflection measured by special markers of the elaborated computer game system are correlated with each other as a valid indicators of a new appropriate diagnostic instrument. The factor of the game type — individual and collaborative — was also controlled to reveal better conditions for manifestation of the higher mental actions level. Another independent variable was abstract intelligence. The results showed that participants exhibit the higher level of the mental actions of analysis and planning in collaborative game. Moreover, the present game type provides the more robust interactions between the mental actions of analysis and planning, and planning and reflection. Furthermore, the strongest patterns of these correlations between the researched variables were demonstrated by those pairs of participants whose intelligence levels were rather equal. Thus, the main hypothesis was confirmed and let derive some important advantages by using the tested computer game system as a potential future diagnostic tool.

Informed Consent from the Participants’

Legal Guardians

Written informed consent to participate in this study was provided by the participants’ legal guardian.

CRediT authorship contribution statement

Arkady Margolis: Conceptualization, Methodology, Supervision. Evgeniya Gavrilova: Investigation, Formal analysis, Writing — original draft, Writing — review & editing. Lev Kuravsky: Conceptualization, Methodology, Supervision. Elena Shepeleva: Investigation, Formal Analysis, Writing — review. Vladimir Voitov: Visualization, Software, Formal analysis. Sergey Erma­kov: Investigation, Writing — review. Pavel Dumin: Formal Analysis.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

 

 

 

1. Adachi, P.J.C, & Willoughby, T. (2013). More than Just Fun and Games: The Longitudinal Relationships Between Strategic Video Games, Self-Reported Problem Solving Skills, and Academic Grades. Youth Adolescence, 42, 1041—1052. doi.org/10.1007/s10964-013-9913-9.
2. Brown, A.L., & Bransford, J. (1982). Learning, Remembering, and Understanding (Technical report No. 244). Illinois University, Urbana Center for the Study of Reading by Bolt, Beranek, & Newman, Inc. Cambridge, MA, 261 p.
3. Chu, M-W, Chiang, A. (2018). Raging Skies: Development of a Digital Game-Based Science Assessment using Evidence-Centered Game Design. Alberta Science education journal. 45, no. 2, 37—47.
4. Davydov, V.V. (1996). Teoriya razvivayushchego obucheniya [The theory of developmental education]. Moscow, 544 p. (In Russian).
5. Elkonin, D.B. (1966). Vozrastne vozmozhnosti usvoeniya znaniy (mladshie klassy shkoly) [Age-related opportunities for the assimilation of knowledge (primary school)]. Moscow, Prosveshchenie [Education], 442 p. (In Russian).
6. Gallagher, D., & Grimm, L.R. (2018). Making an impact: The effects of game making on creativity and spatial processing. Thinking Skills and Creativity 28, 138—149. doi. org/10.1016/j.tsc.2018.05.001.
7. Godwin, K.E., Lomas, D., Koedinger, K.R., Fisher, A.V. (2015). International Journal of Gaming and Computer- Mediated Simulations, 7(4), 18—39. DOI: 10.4018/ IJGCMS.2015100102.
8. Gordeeva, T.O., Sychev, O.A., Sidneva, A.N., & Pshenichniuk, D.V. (2019). Ot chego zavisit zhelanie mladshikh shko’nikov uchit’sya? Structura predmetnoy motivatsii shkol’nikov, obuchayushchiksya v ramkakh raznykh obrazovatel’nykh system [What determines the desire of younger students to learn? The structure of the subject motivation of schoolchildren studying in different educational systems]. Vestnik RFFI. Fundamental’nye problemy v issledovaniyah psihicheskogo zdorov’ya cheloveka i obshchestva [Fundamental problems in the study of men’s and society psychic health] 4 (104). DOI: 10.22204/2410-4639-2019-104-04-31-40. (In Russian).
9. Guruzhapov, V. A. (2000). Ekspertiza uchebnogo protsessa razvivayushchego obucheniya v sisteme D.B. El’konina—V.V. Davydova [Examination of the educational process of developmental education in the system of D.B. Elkonin — V.V. Davydov]. Мoscow, 76 p. (In Russian).
10. Foroughi, C.K., Serraino, C., Parasuraman, R., & Boehm-Davis, D.A. (2016). Can we create a measure of fluid intelligence using Puzzle Creator within Portal 2? Intelligence, 56, 58—64. doi.org/10.1016/j.intell.2016.026.011.
11. Hamari, J., Tuunanen, J. (2014). Player Types: A Meta- synthesis. Transactions of the Digital Games Research Association, 1(2), 29-53. DOI:10.26503/todigra.v1i2.13.
12. Koivisto, J., Hamari J. (2019). The rise of the motivational information systems: A review of gamification research. International Journal of Information Management, 45(191), 210 DOI:10.1016/j.ijinfomgt.2018.10.013.
13. Kritsky, A.G. (1989). Metodika gruppovoy organizatsii uchebnoy deyatel’nosti s ispol’zovaniem komp’utera. Voprosy psikhoogii [Method of the group organization of educational activity with computer. Voprosy psikhologii], 1.
14. Kritsky, A.G., Shcherbinin, M.Yu. (2007) Komp’uternye kommunikatsii v sovmestnoy uchebnoy deyatelnosti. Pedagogicheskaya psikhologiya [Computer communications in collaborative educational activity. Educational psychology], 93 —104.
15. Kuravsky, L.S., Baranov, S.N., & Kornienko, P.A. (2005).   Obuchaemye   mnogofaktornye    seti    Markova    i ikh primenenie dlya issledovaniya psikhologicheskikh kharakteristik [Markov’s multifactor learning networks and their application for studying psychological characteristics]. Neirokomp’yutery: razrabotka i primenenie [Neurocomputers: development and application] 12, 65—76. (In Russian).
16. Leontyev, A.N. (1959). Problemy razvitiya psikhiki [Problems of mental state development]. Moscow: 4th Ed., 1981.
17. Lumsden, J., Skinner, A., Woods, A.T., Lawrence, N.S. & Munafò, M. (2016). The effects of gamelike features and test location on cognitive test performance and participant enjoyment, PeerJ, 4, e2184. http://doi.org/10.7717/ peerj.2184.
18. Margolis, A.A., Kuravsky, L.S., Shepeleva, E.A., Gavrilova, E.V., Petrova, G.A., Voitov, V.K., Yurkevich, V.S., & Ermakov, S.S. (2018). Vozmozhnosti kompyuternoy igry “Plines” kak instrumenta diagnostiki kompleksov kognitivnykh sposobnostey shkol’nikov [Elektronnyi resurs] [Possibilities of the computer game “Plines” as a tool for diagnosing complexes of school children’s cognitive abilities]. Sovremennaya zarubezhnaya psikhologiya [Modern foreign psychology]. 7 (3), 38—52. doi.org/10.17759/jmfp.2018070304. (In Russian).
19. Margolis, A.A., Kuravsky, L.S., Gavrilova, E.V., Shepeleva, E.A., Ermakov, S.S., Voitov, V.K. (2020) Razrabotka komp’uternoy igrovoy sistemy ‘PL-modified’ kak instrumenta diagnostiki universal’nykh uchebnykh deystviy mladshykh shkol’nikov. Tsiphrovaya gumanitaristika i tekhnologii v obrazovanii (DHTE 2020) [Elaboration of ‘PL-modified’ computer game system as a diagnostic tool for measurement of higher mental actions by junior-school students]. [Digital Humanities and Technologies in Education]. Compendium of the All-Russian scientific prachtichal conference with an international participation]. MSUPE, 7—14.
20. Margolis,  A.A.,  Kuravsky,  L.S.,  Artemenkov,  S.L., Voitov, V.K., Shepeleva, E.A., Gavrilova, E.V., Ermakov, S.S. (2020). Analysis of statistical dependencies between intellectual characteristics and gaming ability, measured with the aid of a computer game. International Journal of Modeling, Simulation, and Scientific Computing, 11, 6, 205066-1-22.
21. McClarty, K.L., Orr, A, Frey, P.M., Dolan, R.P., Vassileva, V., McVay, A. (2012). A Literature Review of Gaming in Education. Research Report, Pearson, 36 pages.
22. McDaniel, M.A. (2012). Using quizzes to enhance summative-assessment performance in a Web-based class: An experimental study. Journal of Applied Research in Memory and Cognition, 1(1), 18—26 DOI:10.1016/j.jarmac.2011.10.001
23. Quiroga, M.A., Román, F.J., De La Fuente, J., Privado, J., & Colom, R. (2016). The Measurement of Intelligence in the XXI Century using Video Games. The Spanish Journal of Psychology, 19, 89 p. doi.org/10.1017/sjp.2016.84.
24. Quiroga, M.A., Escorial, S., Roman, F.J., Morillo, D., Jarabo, A., Privado, J., & Colom, R. (2015). Can we reliably measure the general factor of intelligence (g) through commercial video games? Yes, we can! Intelligence, 53, 1—7. doi.org/10.1016/j.intell.2015.08.004.
25. Perret Clermont, A.N. (1991). Rol’ sotsyal’nykh vzaimodeistviy v razvitii intellekta detey [The role of social communication in children’s intelligence development]. Moscow.
26. PISA 2015. Collaborative problem-solving framework, 2017.
27. Polivanova, N.I., Rivina, I.V., Ulanovskaya, I.M. (2013). Vyyavlenie umeniya uchashchikhsya nachal’noy shkoly deistvovat’ sovmestno v usloviyakh sotsyal’no-cognitivnogo konflikta. Psikhologicheskaya nauka i obrazovanie psyedu.ru [Elektronnyi resurs]. [Revealing skills of elementary school students to work together in situation of socio-cognitive conflict][Psychological Science   and   Education   psyedu.ru], 5, 4 (In Russ., аbstr. in Engl.) URL: https://psyjournals.ru/ psyedu_ru/2013/n4/65849.shtml
28. Raven, Dzh.K., Kort, Dzh.Kh., Raven, Dzh. (2002). Rukovodstvo k progressivnym matrictsam Ravena I slovarnym shkalam. Razdel 3. Standartnye progressivnye matritsy Ravena (Vklyuchaya parallelnye I plus versii) [Guide to the Progressive Raven Matrices and Vocabulary Scales: Section 3: Standard Progressive Matrices (including parallel and plus versions)]. Moscow, Kogito-Tsentr, 144 p. (In Russian).
29. Roubtsov, V.V. (1996). Osnovy sotsialno- geneticheskoy psikhologii [Social-genetic psychology framework]. Moscow: Voronezh.
30. Voiskunsky, A.E., Bogacheva, N. (2017). Computer games and creativity: the positive aspects and negative trends. Journal of Modern Foreign Psychology. 6, 4, 29—40 DOI:10.17759/jmfp.2017060403.
31. Vygotskii, L.S. (1935). Umstvennoe razvitie detei v processe obucheniya [Children’s mental development during learning process]. Moscow: L.
32. Zuckerman, G.A. (2016). Igra i uchenie: vstrecha dvukh vedushchikh deyatel’nostei. Kul’turno-istoricheskaya psikhologiya [Play and Learning: How the Two Leading Activities       Meet].        [Cultural-Historical        Psychology], 12, 2, 4—13. (In Russ., аbstr. in Engl.) doi:10.17759/ chp.2016120201
33. Zuckerman, G.A. (2020). Sovmestnoe uchebnoe deistvie: reshennye i nereshennye voprosy. Psikhologicheskaya nauka i obrazovanie [Co-action of Learners: Resolved and Unresolved Issues]. [ Psychological Science and Education], 25, 4, 51—59. (In Russ., аbstr. in Engl.) doi:10.17759/ pse.2020250405.