2022. Vol. 18, no. 2, 13–20
ISSN: 1816-5435 / 2224-8935 (online)
Embodied Cognition in Education: Possibilities and Limitations of Hybrid Representations
Keywords: embodied cognition, learning, cognitive psychology, hybrid representations, grounded cognition, visualization, education
Journal rubric: Theory and Methodology
Article type: scientific article
Funding. The article was written on the basis of the RANEPA state assignment research programme.
For citation: Loginov N.I., Madni A.O., Spiridonov V.F. Embodied Cognition in Education: Possibilities and Limitations of Hybrid Representations. Kul'turno-istoricheskaya psikhologiya = Cultural-Historical Psychology, 2022. Vol. 18, no. 2, pp. 13–20. DOI: 10.17759/chp.2022180202.
Currently there is a large number of theoretical models and empirical facts indicating the important role of the body’s sensorimotor activity in the functioning of cognitive processes (for more details, see: [1; 2; 3]). The approach most commonly known as The Embodied Cognition has taken its right place in the field of fundamental research. And now its proponents are trying to answer the legitimate question of practical application of the obtained results. One of the intriguing areas of the applications would be education, where the problem of students mastering abstract material often arises. This problem is especially common in STEM (Science, technology, engineering, and mathematics) education. It seems that the resources provided by the embodied framework are the most convenient and effective for overcoming such difficulties.
Among the studies in the field of embodied cognition, the most developed framework is called the “Grounded Cognition”. On the one hand, this situation is caused by a large number of proposed heuristic experimental methods (e.g., the switching cost or feature verification paradigms), on the other hand, this direction is theoretically the least radical within all varieties of embodied cognition frameworks, and is much easier to compare with theories and facts derived from more mainstream cognitive research. That is why we would be focusing mostly on the grounded cognition paradigms in this review. The main purpose of this review is to systematize the theoretical and experimental studies in the applied field of STEM-education implemented within this framework, as well as to identify the main opportunities and limitations of the obtained results.
One of the most heuristically valuable ideas in the grounded framework is related to the fundamental feature of mental representations: they do not have to be symbolic and amodal. The computer metaphor suggests that we receive information through our perceptual system in a modal-specific way (visual, auditory, tactile, etc.) and then translate it into an abstract symbolic amodal format arranged like machine code. One of the founders of the Grounded Cognition approach, Lawrence Barsalou, pointed out the scarceness of empirical evidence of the said translation. He proposed the alternative scenario that people do not translate one representation into another, but can use modal-specific representations, enacting both perceptual and motor systems, in the process of processing information of any depth and complexity . The computer metaphor approach does not allow for such a move and considers it meaningless, because the notion that the input and output systems (e.g., keyboard, computer display, etc.) are functionally involved in the operation of the central processor would be absurd. The embodied approach on the other hand attempts to combine these two seemingly irreconcilable positions. Its proponents propose the idea that mental representations are associated with both sensorimotor processes and amodal ones. Thus, such representations are hybrid, that is, they contain both multimodal and abstract symbolic components. This position is based on the following arguments:
1) The classical theories of amodal representationalism assume a rather narrow view of the very nature of mental representations. The research based on such a notion, often substitutes “knowing something” as a simple verbal “name giving” , which is clearly not sufficient.
2) Conceptual representations are limited by context [6; 7; 8], which does not at all correspond to the amodal point of view.
3) Neuronal recycling hypothesis  argues that the idea that the abstract conceptual representations are grounded in perceptual and motor systems proposes the answer to the question of how such complex high-level abstract processes (e.g., natural language, mathematics, etc.) could arise and develop in a very short, from an evolutionary viewpoint, time.
Hybrid representations offer diverse opportunities for improving the efficiency of the learning process, but they are also associated with significant limitations. We will try to demonstrate these on the material of two rapidly developing areas of research — the understanding of symbolic expressions and of graphs and diagrams.
In both of these cases, we can find certain echoes of modern research with the provisions of the cultural-historical theory of Lev Vygotsky. It seems to us that the cited works can be interpreted as quite convincing conceptualizations of the principle of mediation, i.e., the use of diverse cultural means to enhance structurally simpler and genetically earlier mental functions .
Understanding Symbolic Expressions
The groundedness of abstract mental representations in the perceptual system could be most easily demonstrated within the field of mathematics, a field of knowledge with the most abstract content. As an example, consider how people read and understand symbolic expressions such as algebraic equations. An algebraic equation can describe a huge class of specific situations, so its abstractness is undeniable. Moreover, the meaning of an algebraic equation, of course, does not depend on what color or what font it is printed with. However, the perceptual system makes a significant functional contribution to the understanding of this class of symbolic expressions.
An algebraic equation contains abstract and hierarchically ordered relationships between variables, but the form of the equation is closely related to perceptual characteristics which can also be represented and affect the way the equation will be understood. For example, the spatial proximity between variables may be related to the order in which arithmetic operations are performed. The order in which arithmetic operations are performed determines their hierarchy (high-level operators are executed before low-level ones). The addition operation requires the variables and operator to be written in full (p + q), while the multiplication operation allows the shortened version (pq). Both of these examples of mathematical notations do not seem to invoke perceptual elements in the mental representation of an equation, but rather concern certain mathematical conventions. However, this impression is deceptive.
On the material of spatial proximity, an experiment was conducted, in which the participants were required to evaluate the correctness of the equations presented to them. It turned out that they cope with the task worse if the parameter of spatial proximity between variables was not associated with the order of arithmetic operations [11; 12]. For example, if there is less distance between the numbers to be multiplied than between the numbers to be added, then this makes it easier to assess the correctness of the equation and difficult, if the distances are arranged vice versa. In addition, it was found that if adult participants who have a grasp at school level algebra were asked to write an equation by hand, they would write variables closer to each other if they were considered higher operation in terms of operations hierarchy or order (for example, multiplication) than variables associated with the operation of the lower hierarchy level (for example, addition) . The researchers assumed that symbols and mathematical operators would automatically activate spatial relationships. As it turned out in fact: the distance to the left and right of the equal sign to the first character was the maximum .
Judging by the available data, the influence of irrelevant visuospatial information only increases with the growth of expertise . In this study, the authors used an online platform to collect data from about 50,000 Dutch schoolchildren who had to evaluate the correctness of the equations. It was found that high school students have a more pronounced relationship between spatial proximity and the order of arithmetic operations: closely spaced variables were interpreted as priority in terms of the order of operations. This effect is paradoxical, since many researchers of cognitive development have for decades assumed that development proceeds from concrete forms of thinking to more abstract and less material-related forms. However, the results cited indicate that with increasing expertise, people become more sensitive to the spatial organization of algebraic expressions.
One possible explanation is the mechanism of perceptual learning: the perceptual system can be trained so that the distribution of attention represents a mathematical problem in accordance with the decision rules. As an example of one of the studies of the role of attention in solving mathematical equations, one can cite a work where the feature verification paradigm was modified on the basis of algebra . Previously, it was found that checking visual features (for example, color) is easier within one visual grouping than when comparing several features . In a study by Margetis et al., equations like “a * x + b * y” were presented and the color of two neighboring elements changed from black to blue or red (see Fig. 1)
The participants had to state whether the color of these neighboring variables is the same or different. The elements could refer to one arithmetic operation (it might be worth reminding that, according to the rules of arithmetic, multiplication is performed before addition), or the elements could belong to different arithmetic operations. It was assumed that the hierarchical organization of the order of operations in the equation will affect the response time about what color the variables are. In particular, if the variables were connected by a multiplication sign, the reaction time for correct answers should have been faster than for variables connected by addition. And so it happened, but only for the participants with a high level of knowledge in the field of algebra. The results obtained led the authors to the conclusion that the perceptual system plays a functional role in determining the correctness of actions for solving equations, and is not just a channel for obtaining information. In another study, using the method of eye movement registration, it was found that when determining the correctness of an equation, eye movement patterns correspond to the syntactic structure of the equation .
Based on the obtained results, the authors proposed the hypothesis of reassembly of perceptual-motor systems called Rigged Up Perception-Action Systems (RUPAS), and designed to explain how people manage to successfully operate complex sign systems without evolutionarily developed cognitive structures and mechanisms for this . The general idea here is that initially operating with sign systems requires a full set of resource-intensive arbitrarily controlled processes, but as learning progresses, they are automated and replaced by more concise perceptual-motor routines.
Based on this hypothesis and the experimental results obtained, the scientific group of Robert Goldstone developed an interactive system the “Graspable Math” designed for teaching algebra, where students can actively manipulate mathematical operators in real time . Such an approach fundamentally does not link abstract equations with concrete elements like coins, matches, apples, pies, etc. It is assumed that such a system allows us to understand an important thesis: the variables and operators themselves are, in a sense, concrete objects that can be manipulated. Various types of equation transformations are performed by physical actions to change the spatial arrangement of mathematical objects. At the moment, evidence has already been obtained of the effectiveness of teaching algebra using this system [19; 20]. But it is necessary to evaluate the effectiveness of the proposed approach in comparison with the traditional one.
Understanding Graphs and Charts
Another area of application of the ideas of Grounded Cognition is data visualization. In the natural sciences, quite often one has to deal with a visual representation of non-obvious abstract patterns. Therefore, in order to be able to optimize graphs and diagrams, to make them as understandable as possible, it is necessary to study what cognitive processes are involved in solving this problem.
Fig. 1. The example of stimuli for the study of color verification based on the understanding of symbolic expressions (adapted from ). (1) Color verification in the condition of the high-level multiplication operator. (2) Color verification in the condition of the low-level addition operator
One of the research directions in this area is the study of visual routines [21; 22] that are used in the process of reading graphs and which may affect the final interpretation of a particular image. The very fact that these routines influence the interpretation of a graph already suggests that such representations should be grounded in the perceptual system. In one of the works the authors used the eye-tracking method to determine what exactly happens in the process of understanding histograms . The subjects had to compare the bars of the histograms by color or by size. As a result, it was found that when reading this type of graph, people first choose a kind of reference point (in the case of histograms, a specific column on it), with which they compare the rest. It turned out that for histograms of the same color, but different in height, such a reference point is most often the highest bar. And if the columns differ in color, but are the same in size, then it is the darkest. However, if the columns differ both in color and size, then the subjects are guided by the attribute that is relevant to the task (that is, how exactly it is worth comparing the columns — by color or by size). Thus, perceptual templates turn out to be task-specific, and if the presented histograms allow for several options for understanding, then the templates contribute to the interpretation by determining the initial reference point. This result is consistent with others obtained in the course of estimating the number of objects on histograms, where the final interpretation of the graph also depended on the selected reference point .
A whole series of studies was devoted to how, when reading color graphs, people match different colors and concepts that define the semantics of the graph . Evidence has been obtained in favor of the fact that histograms reflecting a different number of objects (fruits) are better understood if the color of the bars matches their color . That is, the perceptual characteristics of the graph affect the accuracy of its interpretation. However, as the authors of this study point out, such an effect occurs only if there is a strong association between the color and the semantics of the category displayed on the graph.
Thus, we can conclude that data visualization is more effective the more it follows the principle of isomorphism: the perceptual characteristics of graphs should somehow correspond to the semantics of the displayed categories.
Another direction of research into the groundedness of mental representations in a perceptual system in the field of visualization is related to the study of the role of students generating schemas that visualize the operation of complex systems. In general, it has already been established that if students independently generate some explanations, examples or analogies to the material being studied, then this increases the effectiveness of learning [27; 28]. But how important is the visual format of these explanations and analogies?
At the moment, there is already evidence in favor of the fact that the explanation of educational material, accompanied by its own schemes and diagrams, is more effective than without them . In particular, it has been found that, when reading a text about tectonic plates, asking students to draw a diagram to accompany the text, they perform better on a follow-up test of similar content than students who were asked to write a short summary of the text they have read . Similar results were obtained for the text on the law of conservation of energy, the understanding of which was better if students were asked to draw a diagram rather than write their own text .
One possible explanation for the advantage of visualizations over verbal descriptions is the need to translate from one representation format to another. Such a translation can help to find gaps and contradictions in the original ideas, as well as their subsequent development and refinement . In general, this kind of translation might help due to a deeper processing of information. However, another explanation is also possible, which suggests that perceptual and verbal representations correspond with each other and, due to such integration, provide an advantage in learning .
However, there is also evidence against the use of visualizations in teaching. In particular, it was found that the very drawing of diagrams to understand the text in physics can lead to specific comprehension errors, when the incorrect spatial arrangement of the components of the diagram leads to additional difficulties in understanding the material being mastered . Yet, the author points out that such errors occur only among beginners who have no experience in using diagrams. Thus, the benefits of visualization in learning are mediated by the experience of using diagrams.
At the same time, a number of studies suggest that spatial abilities can play a role as well. In particular, it was found that people with a low level of such abilities spend more resources on building a visual representation, and people with a high level of spatial abilities willingly spend these resources on matching visual and verbal representations . In addition, people with a low spatial abilities level demonstrate difficulties in the field of animation of mechanical systems (they cannot imagine exactly how a particular device works and answer the corresponding questions) . Besides, it has been found that people with low levels of spatial ability tend to perceive visualizations as static pictures , while people with higher levels of spatial ability embed visualizations into more complex mental representations and manipulate them effectively.
Finally, one of the best-known studies has shown that if subjects are asked to explain or draw how a device (a bicycle pump) works, information about the structural components of the pump is more accurately learned and presented in the case of visualization compared to verbal explanation. No differences were found in responses to the function and mechanics of this pump . Furthermore, the authors indicated that, according to the results of their study, visualization still helps people with low spatial abilities in learning, which is generally consistent with the empirical evidence already accumulated in this area. Thus, we can conclude that it is worth offering a student to draw a diagram or diagram in the process of mastering abstract material if he has sufficiently low spatial abilities, and also if we are talking about the structure, and not about the functions of the phenomenon being studied.
Summing up, we can point to the application potential of using hybrid representations, combining both perceptual components and abstract amodal ones, in an educational context. The very concept of hybrid representations can be a clue to researchers seeking to uncover the mechanisms that underlie the mediation of conceptual thinking, as well as a point of convergence between modern cognitive research and the cultural-historical approach. In particular, the studies described above can be interpreted as illustrations of how various cultural tools in the form of visualizations (graphs or diagrams) and in the form of a system of mathematical symbols affect learning processes.
From the point of view of practical application, the results that testify to the perceptual groundedness of abstract representations can be used in a wide range of educational contexts, ranging from the creation of special software that can suggest to our perceptual system the best ways to assimilate educational material, and ending with local recommendations for working with visualizations and verbal descriptions of complex systems in order to make them more understandable to students.
- Loginov N.I., Spiridonov V.F. Voploshchennoe poznanie (embodied cognition): osnovnye napravleniya issledovanii [Key areas of research in the embodied cognition approach]. Vestnik Sankt-Peterburgskogo universiteta. Psikhologiya i pedagogika = Vestnik SPbSU. Psychology and Education, 2017. Vol. 7, no. 4, pp. 343—364. DOI:10.21638/11701/spbu16.2017.404 (In Russ.).
- Loginov N.I., Spiridonov V.F. Voploshchennoe poznanie kak sovremennyi trend razvitiya kognitivnoi psikhologii [Embodied Cognition as a Current Trend in Cognitive Psychology]. Vestnik Sankt-Peterburgskogo universiteta. Psikhologiya i pedagogika = Vestnik SPbSU. Psychology and Education, 2017. Vol. 7, no. 1, pp. 25—42. DOI:10.21638/11701/spbu16.2017.102 (In Russ.).
- Madni A.O., Spiridonov V.F. Radikal’nyi podkhod k voploshchennomu poznaniyu: teoriya dinamicheskikh system [Radical approach to embodied cognition: dynamic system theory]. Rossiiskii zhurnal kognitivnoi nauki = The Russian Journal of Cognitive Science, 2018. Vol. 5, no. 3, pp. 37—55. (In Russ..
- Barsalou L. W. Perceptions of perceptual symbols. Behavioral and brain sciences, 1999. Vol. 22, no. 4, pp. 637—660. DOI: 10.1017/S0140525X99532147
- Mahon B.Z., Caramazza A. Why does lexical selection have to be so hard? Comment on Abdel Rahman and Melinger’s swinging lexical network proposal. Language and Cognitive Processes, 2009. Vol. 24, no. 5, pp. 735—748. DOI:10.1080/01690960802597276
- Machery E. Doing without concepts. Oxford University Press, 2009. 283 p
- Connell L., Lynott D. Principles of representation: Why you can’t represent the same concept twice. Topics in Cognitive Science, 2014. Vol. 6, no. 3, pp. 390—406. DOI:10.1111/tops.12097
- Yee E., Thompson-Schill S. L. Putting concepts into context. Psychonomic bulletin & review, 2016. Vol. 23, no. 4, pp. 1015—1027. DOI:10.3758/s13423-015-0948-7
- Dehaene S., Cohen L. Cultural recycling of cortical maps. Neuron, 2007. Vol. 56, no. 2, pp. 384—398. DOI:10.1016/j.neuron.2007.10.004
- Vygotskii L.S. Istoriya razvitiya vysshih psihicheskih funkcij [The History of the Development of Higher Mental Functions]. Sobranie sochinenij = Collected works. Moscow, 1983. Vol. 3, pp. 5—329 (In Russ.).
- Landy D., Goldstone R. L. Formal notations are diagrams: Evidence from a production task. Memory & cognition, 2007. Vol. 35, no. 8, pp. 2033—2040. DOI:10.3758/BF03192935
- Rivera J., Garrigan P. Persistent perceptual grouping effects in the evaluation of simple arithmetic expressions. Memory & cognition, 2016. Vol. 44, no. 5, pp. 750—761. DOI:10.3758/s13421-016-0593-z
- Landy D., Goldstone R. L. How abstract is symbolic thought? Journal of Experimental Psychology: Learning, Memory, and Cognition, 2007. Vol. 33, no. 4, pp. 720. DOI:10.1037/0278-73220.127.116.110
- Braithwaite D. W. et al. Non-formal mechanisms in mathematical cognitive development: The case of arithmetic. Cognition, 2016. Vol. 149, pp. 40—55. DOI:10.1016/j.cognition.2016.01.004
- Marghetis T., Landy D., Goldstone R. L. Mastering algebra retrains the visual system to perceive hierarchical structure in equations. Cognitive research: principles and implications, 2016. Vol. 1, no. 1, pp. 25. DOI:10.1186/s41235-016-0020-9
- Duncan J. Selective attention and the organization of visual information. Journal of experimental psychology: General, 1984. Vol. 113, no. 4, pp. 501. DOI:10.1037/0096-3418.104.22.1681
- Schneider E. et al. Eye gaze reveals a fast, parallel extraction of the syntax of arithmetic formulas. Cognition, 2012. Vol. 125, no. 3, pp. 475—490. DOI:10.1016/j.cognition.2012.06.015
- Goldstone R. L. et al. Adapting perception, action, and technology for mathematical reasoning. Current Directions in Psychological Science, 2017. Vol. 26, no. 5, pp. 434—441. DOI:10.1177/0963721417704888
- Ottmar E. et al. Graspable mathematics: Using perceptual learning technology to discover algebraic notation. Integrating touch-enabled and mobile devices into contemporary mathematics education. IGI Global, 2015, pp. 24—48.
- Ottmar E., Landy D. Concreteness fading of algebraic instruction: Effects on learning. Journal of the Learning Sciences, 2017. Vol. 26, no. 1, pp. 51—78. DOI:10.1080/10508406.2016.1250212
- Cavanagh P. Attention routines and the architecture of selection. Cognitive Neuroscience of Attention. New York: Guilford Press, 2004, pp. 13—28.
- Ullman, S. Visual routines. Cognition, 1984. Vol. 18, pp. 97—159. DOI:10.1016/B978-0-08-051581-6.50035-0
- Michal A.L., Franconeri S. L. Visual routines are associated with specific graph interpretations. Cognitive research: principles and implications, 2017. Vol. 2, no. 1, pp. 1—10. DOI:10.1186/s41235-017-0059-2
- Michal A.L. et al. Visual routines for extracting magnitude relations. Psychonomic bulletin & review, 2016. Vol. 23, no. 6, pp. 1802—1809. DOI:10.3758/s13423-016-1047-0
- Schloss K.B. et al. Color inference in visual communication: the meaning of colors in recycling. Cognitive research: principles and implications, 2018. Vol. 3, no. 1, pp. 1—17. DOI:10.1186/s41235-018-0090-y
- Lin S. et al. Selecting semantically resonant colors for data visualization. Computer Graphics Forum. Oxford, UK: Blackwell Publishing Ltd, 2013. Vol. 32, no. ¾, pp 401—410. DOI:10.1111/cgf.12127
- Chi M.T.H. Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in cognitive science, 2009. Vol. 1, no. 1, pp. 73—105. DOI:10.1111/j.1756-8765.2008.01005.x
- Hall V. C., Bailey J., Tillman C. Can student-generated illustrations be worth ten thousand words? Journal of Educational Psychology, 1997. Vol. 89, no. 4, pp. 677. DOI:10.1037/0022-0622.214.171.1247
- Ainsworth S., Loizou Th. A. The effects of self-explaining when learning with text or diagrams. Cognitive science, 2003. Vol. 27, no. 4, pp. 669—681. DOI:10.1207/s15516709cog2704_5
- Gobert J. D., Clement J. J. Effects of student-generated diagrams versus student-generated summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 1999. Vol. 36, no. 1, pp. 39—53. DOI:10.1002/(SICI)1098-2736(199901)36:1<39::AID-TEA4>3.0.CO;2-I
- Edens K. M., Potter E. Using descriptive drawings as a conceptual change strategy in elementary science. School science and mathematics, 2003. Vol. 103, no. 3, pp. 135—144. DOI:10.1111/j.1949-8594.2003.tb18230.x
- Ainsworth S., Bibby P., Wood D. Examining the effects of different multiple representational systems in learning primary mathematics. The Journal of the Learning Sciences, 2002. Vol. 11, no. 1, pp. 25—61. DOI:10.1207/S15327809JLS1101_2
- Aleven V. A., Koedinger K. R. An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. Cognitive science, 2002. Vol. 26, no. 2, pp. 147—179. DOI:10.1207/s15516709cog2602_1
- Wilkin B. Learning from explanations: Diagrams can “inhibit” the self-explanation effect. Reasoning with diagrammatic representations II: Papers from AAAI Fall Symposium, 1997, pp. 136—143.
- Mayer R. E., Sims V. K. For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. Journal of educational psychology, 1994. Vol. 86, no. 3, pp. 389. DOI:10.1037/0022-06126.96.36.1999
- Hegarty M., Just M. A. Constructing mental models of machines from text and diagrams. Journal of memory and language, 1993. Vol. 32, no. 6, pp. 717—742. DOI:10.1006/jmla.1993.1036
- Kozhevnikov M., Hegarty M., Mayer R. E. Revising the visualizer-verbalizer dimension: Evidence for two types of visualizers. Cognition and instruction, 2002. Vol. 20, no. 1, pp. 47—77. DOI:10.1207/S1532690XCI2001_3
- Bobek E., Tversky B. Creating visual explanations improves learning. Cognitive Research: Principles and Implications, 2016. Vol. 1, no. 1, pp. 27. DOI:10.1186/s41235-016-0031-6
Information About the Authors
Previous month: 15
Current month: 3
Previous month: 6
Current month: 1