Making statistics meaningful for psychologists: the merits of a competence-based approach

В статье рассматриваются вопросы, посвященные использованию методов математической статистики при подготовке студентов-психологов. Отмечается, что традиционный подход, согласно которому студентам даются различные «отрывочные», иногда не связанные между собой сведения о математической статистике (часто не специалистами не имеющими отношения к психологии) зачастую не приносит должных результатов: необходимые знания усваиваются недостаточно хорошо, а у самих студентов снижается мотивация освоения данных курсов, что сказывается на недостаточности общей профессиональной компетенции... Вашему вниманию представлен оригинал статьи на английском языке!

... To stop this devastating process and to enhance students’ attitudes (in doing research and valuing it) and to motivate students to behave like a scientist-practitioner, we have decided to shift the helm in training research skills (including quantitative skills). Students will be trained in doing research from the very beginning of their study. With this ‘whole-task approach’ (Van Merriënboer, 1997) the training focuses on the complete complex cognitive skill of doing research (from simple to complex versions). This approach emphasizes the coordination and integration of the constituent skills from the very beginning, and stresses that students should quickly develop a holistic view of the whole task that is gradually embellished and detailed during instruction. The holistic view implies that students have to grasp the necessity to align the successive steps systematically. Students should be stimulated to develop awareness of the inextricable connectedness of the subsequent steps. So within the teaching design emphasis must be put on general methodological principles and theories, and students have to be stimulated to experience and learn the complete research process with all relations and connections between the theoretical, methodological, and statistical levels.

Constructivism holds that the acquisition of complex skills is context-dependent and occurs most effectively in relevant contexts (Lovett & Greenhouse, 2000). The learning environment must provide realistic situations where learning through meaningful practices takes place. By doing research in a great variety of situations, students are more likely assured of a better transfer of the academic subject (Van Merriënboer, 1997). Computer technology and timesaving computer programs make such an approach feasible. ...

Although the comparison is still in its infancy – we could compare only a small part of both curricula in research training – the effects of the competence-based approach are promising. It strengthens our idea that we are on the right track. This study provides support for the claim that integrated, research-based teaching designs can optimize learning outcomes in statistics. The new teaching context removes from statistics its perception as a discipline with a narrow, mathematically focused and anxiety-provoking perspective and nests it in a research-embedded context, related to the psychology domain that triggers student interest and commitment, and as a consequence, facilitates the study process. Results are promising!

---

To be able to behave like a scientist–practitioner in the fields of professional psychology by which graduated students based their professional practices on scientific findings and scientifically sound theories, students are trained to do (psychological) research. Traditionally the research training in the psychology curricula consists of a number of service courses in research methods and statistics which are given individually and independently from each other by specialists who are not interested in clinical research applications or are indifferent to students’ needs and feelings about the subject matter. Such a research training practice has some profound deficiencies (Gelso, 2006). Especially in the field of statistics education students face many problems (Garfield & Ben-Zvi, 2007) that retard students’ investment in research. By turning away from doing research and valuing it, the final result is that – even if students don’t want to be a researcher - professional psychology loses a chance to get more insights derived from clinical practices.

To stop this devastating process and to enhance students’ attitudes (in doing research and valuing it) and to motivate students to behave like a scientist-practitioner, we have decided to shift the helm in training research skills (including quantitative skills). Students will be trained in doing research from the very beginning of their study. With this ‘whole-task approach’ (Van Merriënboer, 1997) the training focuses on the complete complex cognitive skill of doing research (from simple to complex versions). This approach emphasizes the coordination and integration of the constituent skills from the very beginning, and stresses that students should quickly develop a holistic view of the whole task that is gradually embellished and detailed during instruction. The holistic view implies that students have to grasp the necessity to align the successive steps systematically. Students should be stimulated to develop awareness of the inextricable connectedness of the subsequent steps. So within the teaching design emphasis must be put on general methodological principles and theories, and students have to be stimulated to experience and learn the complete research process with all relations and connections between the theoretical, methodological, and statistical levels.

Constructivism holds that the acquisition of complex skills is context-dependent and occurs most effectively in relevant contexts (Lovett & Greenhouse, 2000). The learning environment must provide realistic situations where learning through meaningful practices takes place. By doing research in a great variety of situations, students are more likely assured of a better transfer of the academic subject (Van Merriënboer, 1997). Computer technology and timesaving computer programs make such an approach feasible.

Such a transformation requires that faculty specialists (i.c. methodologists and statisticians) give up their autonomy and work cooperatively in a team. In the redesign of the research curriculum, the books and their contents of the individual subjects are no longer central, but according to the constructivist paradigm, the focus is aligned to (research) assignments. To emphasize the difference from the earlier curriculum with its separate research methods and statistics courses, in the case of the integrated courses in the new curriculum we will speak about competences. In this case we are concerned with the bachelor psychologist’s research competence that features statistical knowledge and skills as vital constituents.

A research competence-based curriculum has been designed to replace seven service courses of each comprising 120 hours of study time: two statistics courses, three courses about research methods; one course on applications of the computer program SPSS; and one course on literature research. Including the bachelor thesis (240 hours), a total of 1080 hours could be rearranged and reallocated. 

 The rearrangement and reallocation of the service courses into the new curriculum consisting of seven research practicals is specified in Table 1. This table shows that in order to make empirical investigations manageable the research process has been broken down according to the research stages and the rules of the American Psychological Association (APA) for journal articles (Introduction (‘Problem stage’), Methods, Results, Discussion, the last ‘stage’ is omitted in the table). The Methods and the Results columns of the table show the contents of the two subjects that were formerly taught separately: research methods and statistics respectively. Each practical contains at least three different research assignments. The first assignment is a worked-out example in order to reduce the ‘mental’ or cognitive load (Lovett & Greenhouse, 2000); the second a less worked-out assignment (fill-in) where students have to contribute more, and the final assignment is the task where students have to carry out their own research in accordance with the learning goals. Within each practical the intensity of the coaching decreases and the contribution of the individual student increases (scaffolding), according to the Four Component / Instructional Design (4C/ID) model (Van Merriënboer, 1997). Not all elements of the research process are emphasized equally in the practicals. In the first practical, called Parametric Data-Analysis, the Results stage is emphasized, which means that although examples and cases of complete investigations are given, most attention is given to the data analysis. In the second practical attention shifts to the second stage of the research process (Methods). Now we expect students to apply correctly the statistics knowledge and skills acquired (in this case regression and correlation), and to be competent in extending this ‘prior’ statistical knowledge and skills to several specific features (moderation, mediation). We should point out that in their bachelor’s education students now get acquainted with statistical techniques and aspects of statistical thinking in far more opportunities than had been the case when students received only their two service courses. Note that in the first and second practicals we limited the statistics to the mean, standard deviation and extensions of these to analysis of variance and regression/correlation. As a consequence students can get a better understanding of the mean and its characteristics, without confusing these with other (e.g. non-parametric) statistics. In addition, all the statistics material learned comes back in all later research assignments. Thus students discover the necessity, functionality and utility of statistics, through experiencing them, and become more motivated as a result of real data and task authenticity. Finally, their knowledge of statistics should be interiorized properly by repeatedly performing data analysis; their knowledge becomes transferable, and deep understanding should grow as a consequence of the use of a variety of (psychological) cases (Van Merriënboer, 1997; Lovett & Greenhouse, 2000).

It has taken several years to gain insights into the consequences of a curriculum redesign, to evaluate the definitive psychological research programme, and to select the appropriate delivery, sequence and content of statistical topics in every practical. During this period of tryouts students’ motivation, attitude, uses of self-regulating learning strategies (deep learning or superficial learning and their components) and learning outcomes have been monitored. Based on the positive results in the quasi-experiments, Open Universiteit Nederland decided in 2003 to change the curriculum of statistics and research methods definitively. At this moment, however, the whole curriculum has not yet been redesigned. In September 2004 the last opportunity for students to complete the classic statistics service courses was discontinued and research practical 1 (Parametric data analysis) was officially introduced. In 2009 the whole curriculum will be designed.

To align learning goals, instruction, and assessment, the examination in each practical comprises an (empirical) investigation. So statistical (and methodological) knowledge and skills will be tested indirectly.

In this paper we focus on the effects of the two different curricula – traditional service courses and competence-based - on learning outcomes: are student performances in statistical problem solving in an integrated curriculum design better than in the case of individual classic statistics courses?  By student performances we do not only mean learning outcomes, but also attitudes, motivation and cognitive strategies. Figure 1 reflects Zusho and Pintrich’s model of motivation and self-regulated learning (2003), supplemented with attitude toward statistics.

In the Presage stage two ‘actors’ are distinguished: the students and their characteristics and the teaching and learning environment. In this paper we highlight the teaching context, activating the targeted determinants for learning activities in the Process stage. In the Process stage we distinguish attitude, motivation, and cognitieve and meta-cognitive strategies.

• Attitude toward statistics

Attitudes originate from social psychology (Smith & Mackie, 2000) as fundamentals for constructs like social categorisation and social identity. Attitudes mirrors influences from the social environment on the individual, reflected in automatic evaluations of situations. Eagly and Chaiken (1993) define attitudes as ‘psychological tendencies, expressed by evaluating particular entities with some degree of favour or disfavour’. They consider attitudes as well established subsystem of cognitive schemas, having a considerable impact on behavioural and affective responses, especially selective perception. Attitudes protect the ego against uncomfortable reality and organize and simplify experiences, expressing self-concept and personalised values (Eagly & Chaiken, 1993). Socialisation processes, including academic socialisation (Donald, 2002), affect the construction of attitudes. In their impact on behaviour, attitudes are more automatic and subconscious, and are assumed to act via worn paths in subsystems of cognitive schemas. As a consequence attitudes are less flexible and less sensitive to direct cognitive regulation, because they act from an established sublevel in the cognitive schemes and show a tendency to outperform the coordinating cognitive schemes. They automatically influence series of general behaviour rather than affecting specific behaviour (Eagly & Chaiken, 1993). Here we see an analogy with conditioned responses from behaviourism. Eagly and Chaiken (1993) consider attitudes to be related to selective perception. A heritage of behaviourism though is to neglect the information processes involved in learning. In the statistics domain attitudes toward statistics are studied and related to statistics anxiety or achievement. Anxiety negatively influences academic achievement. Schau, Stevens, Dauphinee and Del Vecchio (1995) in their Survey of Attitude Towards Statistics (SATS) distinguish four attitudinal dimensions, which are of interest in studying statistics: Affect, which reflects affective evaluations of statistics; Cognitive Competence, representing opinions about the cognitive skills needed to master statistics; Task Value (statistics), which is an evaluation of the usefulness and importance of statistics and Difficulty, reflecting beliefs about the problems that will be met in studying statistics.

• Motivation

Pintrich et al. (2002) define motivation as ‘the process whereby goal-directed activity is instigated and sustained’, reflecting a dynamic process between individual and context. In a revised Expectancy Value model (see e.g. Eccles & Wigfield, 2002) Pintrich et al. relate motivational dimensions to learning strategies. Biggs (2003) endorses that teaching activities and tasks have to stimulate students’ interest (Renninger, Hidi & Krapp, 1992), which is related to cognitive engagement and the cognitive strategies that are used (CSU’s, Pintrich, Smith, Garcia, & McKeachie, 1991, 1993; Lewalter & Krapp, 2004).

Intrinsic Value is a mastery oriented task evaluation, characterised by positive affect and related to interest, CSU’s and Self-Regulation (Pintrich et al., 1991, 1993). It represents the cognitive and affective reasons for being engaged in a task, reflecting knowledge, positive affect and appreciation of the task (Csikszentmihalyi, 1990; Renninger et al., 1992).

Task Value Research is a student’s evaluation of the importance and usefulness of the research task, related to more distant or instrumental goals like e.g. the future profession (Eccles et al., 2002; Pintrich et al., 1991, 1993).

Test Anxiety is ‘ set of phenomenological, physiological and behavioural responses that accompanies concern about possible negative consequences or failure on an exam or similar evaluative situation’ (Zeidner, 1998). It relates to avoidance goals and occurs in specific domains, like mathematics and perhaps statistics. Test Anxiety includes a cognitive and affective component. Empirical studies have shown that the worry (cognitive) component in particular is closely linked to performance decrement (Covington, 1993).

The expectancy component in the EV model is reflected by Self-Efficacy, which has been defined as ‘people’s judgement of their capabilities to organize and execute courses of action required to attain designated types of performance’ (see Pintrich et al., 2002). Self-Efficacy is goal-directed and related to behavioural engagement, reflected by effort and persistence. Compared with other self-concepts Self-Efficacy is the most contextual defined and domain related construct, it varies as a function of personal and environmental differences and shows little generalization across areas (Pintrich et al., 2002).

Based on literature regarding motivations and attitudes we can consider both psychological concepts as different in their qualities and effects. We can infer that motivations are intra-individual, multi-dimensional dynamic processes constituted by the individual’s interactions with the environment, closely linked to cognitions (goals) and affect and that they can be consciously controlled and regulated by the individual himself. For learning processes motivations are elementary conditions to start and sustain learning. Attitudes act like conditioned evaluations and beliefs.

• Cognitive and self-regulated strategies

In terms of cognitive processes, we are mainly concerned with students’ selfreported use of cognitive and self-regulated strategies. Strategies can be divided into two main categories: superficial strategies that only require surface level processing, and those strategies that require more deeper processing of course material. Generally, researchers have shown that it is more adaptive to use deeper processing strategies, in terms of long-term retrieval of information (Pintrich 2000; Pintrich and Schrauben 1992). In this study, we examine three cognitive strategies: (1) rehearsal, a surface level strategy, where students focus on memorizing and recall of facts; (2) elaboration, a deeper processing strategy, where students focus on extracting meaning, summarizing, or paraphrasing, and (3) critical thinking, another

deeper processing strategy where students apply knowledge and domain-related criteria in a variety of situations in which problem solving, decision-making and critical evaluations are required (Donald, 2002; Pintrich et al., 1991, 1993). It is a well reasoned, substantiated and questioning strategy, investigating assumptions and seeking for evidence. Critical Thinking encompasses logical thinking, problem solving and abstracting (Donald, 2002). Finally, Self-Regulation is a meta-cognitive strategy, activating prior knowledge and monitoring, planning and regulating the cognitive learning strategies that are used (Pintrich et al., 1991, 1993).

Methods

• Participants

Data were collected from 468 psychology students in distance education, in two different samples: 340 students who participated in the traditional statistics service courses, and 128 students who have been confronted with the whole-task teaching design. In both samples approximately 5% of the participants were male and 95% female, which is an adequate reflection of the total population of Psychology students at the Dutch Open University.

• Procedures

Students were invited to participate in the study via e-mail or a letter two weeks before the questionnaire was presented. Participation was voluntary, i.e. it did not carry any rewards. The questionnaire was offered via a closed website or by letter for participants without an e-mail address. After a month students received a reminder to stimulate them to submit the questionnaire.

• Materials

Dutch translations of the SATS (Schau et al., 1995) and the MSLQ (Pintrich et al., 1991, 1993) were used to assess attitude, motivation and cognitive strategies, using 1 to 5 points Likert scales, whereby 1 indicates ‘ don’t agree at all’ and 5 ‘I totally agree’. For the statistics performance test, seven open questions and one closed question were developed (for examples see for instance Reading, 1996). The answers and arguments on the statistics questions were evaluated and classified in accordance with the Structure of Observed Learning Outcomes (SOLO) taxonomy (Biggs & Collis, 1982). For Autonomy and Dependency Likert type items of the Big Five Scale (Goldberg, 1990) have been used.

All scales (including the statistics performance test) were analysed by the Rasch model for one dimensionality, misfits, and person and item reliability, followed by Structural Equation Modelling of the Rasch scales.

Results

Raschmodel scores of all scales range from level -4 till +4 logits with a mean of 0. A one-way between groups MANOVA was performed to investigate differences in attitude, motivation, cognitive strategies and learning outcomes. There was a statistical significant difference between the two teaching contexts, F (15, 452) = 4.616, p = .000; Wilks’ Lambda = .867 and partial eta squared = .13. When the results for the dependent variables were considered separately, all effects except one (rehearsal) were in the predicted direction in favour of the competence-based approach (see Table 2).

Discussion and conclusion

Although the comparison is still in its infancy – we could compare only a small part of both curricula in research training – the effects of the competence-based approach are promising. It strengthens our idea that we are on the right track. This study provides support for the claim that integrated, research-based teaching designs can optimize learning outcomes in statistics. The new teaching context removes from statistics its perception as a discipline with a narrow, mathematically focused and anxiety-provoking perspective and nests it in a research-embedded context, related to the psychology domain that triggers student interest and commitment, and as a consequence, facilitates the study process. Results are promising!

1. Barab S.A., Squire K.D., & Dueber W. (2000). A co-evolutionary model for supporting the emergence of authenticity. The Journal of Education Technology, Research, and Development, 48(2), 37–62.
2. Bijker M.M., Wynants G., & Van Buuren J.A. (2006). A comparative study of the effects of motivational and affective factors on studying statistics. Paper presented at ICOTS7, Salvador, Brazil.
3. Biggs J.B., & Collis K.F. (1982). Evaluating the Quality of Learning: the SOLO taxonomy. New York: Academic Press.
4. Bijker M.M. (2006). Methodological issues: the paradigm change from Likert scales to Rasch scales:  Measuring the effects of a new teaching and learning design in statistics. Unpublished thesis. Heerlen, Open Universiteit.
5. Covington M.C. (1993). Making the Grade. A self-worth perspective on motivation and school reform (2 ed.). Cambridge: University Press.
6. Czikszentmihalyi M. (1990). Flow: the classic work on how to achieve happiness. London: Random House Group Ltd.
7. Donald J.G. (2002). Learning to think. San Francisco: Jossey-Bass.
8. Eagly A.H., & Chaiken S. (1993). The Psychology of Attitudes. Orlando: Harbourt Brace College Pubishers.
9. Eccles J.S., & Wigfield A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 109-132.
10. Garfield J., & Ben-Zvi D. (2007). How Students Learn Statistics Revisited:A Current Review of Research on Teaching and Learning Statistics. International Statistical Review, 75(3), 372-396.
11. Gelso C.J. (2006). On the Making of a Scientist–Practitioner: A Theory of Research Training in Professional Psychology. Training and Education in Professional Psychology, S(1), 3-16.
12. Goldberg L.R. (1990). An alternative description of personality: the Big Five Factor Structure. Journal of Personality and Social Psychology, 59(10), 1216-1229.
13. Lewalter D., & Krapp A. (2004). The role of contextual conditions of vocational education for motivational orientations and emotional experiences. European Psychologist, 9(4), 210-221.
14. Lovett M.C., & Greenhouse J.B. (2000). Applying cognitive theory to statistics instruction. The American Statistician, 54(3), 196-206.
15. Pintrich P.R., Smith D.A.F., Garcia T., & McKeachie W.J. (1991). A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ).Unpublished manuscript, Ann Arbor - Michigan: University of Michigan.
16. Pintrich P.R., Smith D.A.F., Garcia T., & McKeachie W.J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801-813.
17. Pintrich P.R., & Schunk D.H. (2002). Motivation in Education (2 ed.). New Jersey: Merrill Prentice Hall.
18. Renninger K.A., Hidi S., & Krapp A. (1992). The Role of Interest in Learning and Development. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc.
19. Schau C.G., Stevens J., Dauphinee T.L., & Del Vecchio A. (1995). The development and validation of the survey of attitudes toward statistics. Educational and Psychological Measurement, 55(5), 868-875.
20. Smith E.R., & Mackie D.M. (2000). Social Psychology (2 ed.). Philadelphia: Psychology Press.
21. Van Buuren J.A. (2006). Teaching statistics and research methods: an integrated approach. Paper presented at ICOTS7, Salvador, Brazil.
22. Van Buuren J.A. (2008). From subject-oriented to competence-based statistics education. An intervention study in a school of psychology. Heerlen: Open Universiteit Nederland
23. Van Merriënboer J.J.G. (Ed.). (1997). Training Complex Cognitive Skills: a Four-Component Instructional Design Model for Technical Training. New Jersey: Englewood Cliffs.
24. Watson J. M., & Callingham R.A. (2003). Statistical literacy: a complex hierarchical construct. Statistics Education Journal, 2(2), 3-46.
25. Zeidner M. (1998). Test anxiety: The state of the art. New York: Plenum
26. Zusho A., & Pintrich P.R. (2003). Skill and will: the role of motivation and cognition in the learning of college chemistry. International Journal of Science Education, 25(9), 1081 - 1094.