ISSA Proceedings 2014 ~ Current Trends In Educational Research On Argumentation. What Comes After Toulmin?
Abstract: Although many education researchers exclusively use Toulmin´s model, more and more scholars opt for other Informal logic tools, such as dialogue models or argumentation schemes. The present paper describes this tendency of slowly moving from Toulmin to other models and gives a narrow focus to those articles that use other argument models than Toulmin´s to analyze and assess students and/or teachers´ arguments. As a final contribution, we provide a taxonomy of argumentation tools used in educational research in relation to a number of variables such as type of task, age of participants, disciplinary subject, and main skills assessed as significant.
Keywords: argument analysis and assessment, education, skills, taxonomy, tools, Toulmin.
Since the early beginning of the informal-plausible logic theories as a counter-balance to the existing formal-deductive ones, scholars from the informal logic field have made suggestions on how argumentation should be instructed, or what is important when teaching argumentation (e.g. Voss & Means, 1991; Voss, Perkins, & Segal, 1991). In its almost 50 years of existence, if we consider Kahane´s ¨Logic and Contemporary Rhetoric” (1971) as the first official informal logic manuscript as proposed by Johnson (2000), informal logic has expanded into many schools of thought, especially regarding how informal arguments should be analyzed and assessed. Among them, we distinguish the following for their applicability in education research and practice: dialogue analysis, which focuses on argument as a dialogical process taking place in a specific dialogue context in which participants make a series of “moves” forming strategic sequences or even a dialogue game (e.g. Felton & Kuhn, 2001; De Vries, Lund, & Baker, 2002; Felton, 2004; Mcalister, Ravenscroft, & Scanlon, 2004; Prakken, 2005); argumentation schemes, which is a device that aims to formalize (in the sense of giving structure to) everyday arguments mostly related to plausible reasoning (e.g. Walton, 1996; van Eemeren & Grootendorst, 2004; Walton, Reed, & Macagno, 2008; Rigotti & Morasso, 2010); and dialectics and pragmadialectics, which apply in conflict argumentative contexts where one of the opposing views is more sound or acceptable than the other(s) (e.g. Barth & Krabbe, 1982; Walton & Krabbe, 1995; van Eemeren & Grootendorst, 2004).
Notwithstanding the plethora of informal logic tools and methods of analysis and assessment proposed, there is a remarkable tendency among educational researchers to apply Toulmin´s Argumentation Pattern (TAP) as it was proposed in the late fifties (Toulmin, 1958). TAP´s main original contribution was to oppose to the mathematical standards in logic at the time. From an informal logic perspective, TAP is still considered an acceptable method, as several contemporary authors have successfully addressed questions that Toulmin´s model had provoked (see Hitchcock & Verheij, 2005, for an overview). In the field of education, some of the reasons for its predominance are the following: a) its strong connection with science and scientific reasoning (Duschl & Osborne, 2002); b) its success in coding large data protocols (Voss, 2005); and c) its easiness to use as a measurement of both teaching and learning performance (Erduran, Simon, & Osborne, 2004). On the other hand, TAP also received several criticisms, such as: a) the model concentrates on the proponent (Leitao, 2000), b) it can be difficult to structure reasoning in real time (Simon, Erduran, & Osborne, 2006), c) we should study argumentation in a more holistic and emergent manner rather than imposing an existing analytical pattern such as TAP (Sampson & Clark, 2009), and d) the scheme is restricted to short arguments and the categories impose ambiguities (Kelly, Druker, & Chen, 1998).
The goal of this paper is to provide a reflective overview of educational research in argumentation, and to identify any existing trends in terms of tendencies of preference towards one tool of argument analysis and assessment or another. More precisely, we seek answers to the following questions:
1. Is there a correlation between the use of one tool of argument analysis and assessment or another and the study´s characteristics such as date, design, focus, and variables?
2. Can we discuss about an evolution in the tendency of educational research in the field of argumentation skills?
The data collection was based on both preliminary and secondary sources. Among the first, the databases of Wiley, ScienceDirect, and Springer were used. We also identified some primary journals focusing on the subject of argument and education. Among the secondary sources, three reviews were of special interest to our goal and questions, namely Clark, Sampson, Weinberger & Erkens (2007), Nussbaum (2011), and Rapanta, Garcia-Mila & Gilabert (2013). Further details for both types of sources are provided below.
The criteria for preliminary articles´ selection were the following:
a. Peer-reviewed articles published in scientific journals related to argument and/or education with an explicit focus on argument skills and their enhancement or analysis through concrete interventions and/or methods which lead to the presentation of concrete results regarding the quality of arguing of different age participants in the field of education;
b. Clear dependent variables assessed in sufficient ways and related either to concrete independent variables, in case the study is experimental, or to influencing factors or task characteristics in case the study is non-experimental;
c. Exclusion of studies entirely focusing on outcomes of arguing such as collaborative learning or problem solving.
Finally, we reviewed 73 scientific articles published from 1985 to 2013 with an explicit focus on argument skills enhancement in education. The goal of the review was to give answers to the two research questions mentioned in the introduction.
Three secondary sources were also used in order to better contextualize the observations based on the preliminary review results presented in the next section. Clark et al. (2007) reviewed frameworks for analyzing online dialogical argumentation, and they came up with five most predominant types of focus regarding argument analysis:
a) formal argumentation structure,
b) conceptual quality,
c) nature and function of contributions within the dialogue,
d) epistemic nature of reasoning, and
e) argumentation sequences and interaction patterns.
Toulmin (1958) appears as contributing only to the first focus-criterion of argument analysis and assessment. Rapanta et al (2013) reviewed 97 studies in order to describe how argumentative competence is conceived by educational researchers. They propose three general conceptions of argument, as form, as strategy, and as goal, and three main levels of argument assessment, namely metacognitive, metastrategic and epistemological. Among the findings, TAP is used by more than two thirds of the total studies that define argument as form and mainly when the task is written argumentation, showing its strong connection with structural elements of argumentation. On the other hand, when the focus of the study is argument strategy or goals, TAP is hardly used. Last but not least, in his recent essay Nussbaum (2011) argues that other frameworks of argumentation rather than Toulmin’s may be used for both research and practice in education, such as Walton´s dialogue theory and Bayesian models of everyday arguments. Both tools provide a detailed focus on important aspects of in-class argumentation, such as plausibility and dialecticity. Yet, their application in educational research is still stunted.
The secondary sources helped us identify the main characteristics of the preliminary sources, in terms of the following:
a. Study design, meaning if it is experimental or non-experimental;
b. Type of task, including production (written, oral, semi-oral) or assessment tasks (e.g. fallacies identification, test on argument understanding);
c. Age of participants, meaning children, adolescents, or adults;
d. Subject of argumentation, ranging from natural sciences and mathematics to social/socio-scientific issues and general/everyday issues;
e. Independent variables, including age/grade, prior knowledge, gender, and epistemological knowledge (knowledge about argumentation) of participants, and also task characteristics such as goal instructions, topic, explicit argument teaching as an intervention, or use of diagrams and computer scaffolds;
f. Dependent variables, focusing mainly on quantity and quality of grounds and reasons, quantity and quality of counterarguments and rebuttals, argument strategy, epistemic strength, and quality of argumentative interaction.
Not to our surprise, of the 73 studies reviewed, 30 (41 %) use TAP as their main tool of argument analysis and assessment. However, nearly the same percentage of studies (36 %) uses other informal logic tools, such as interaction analysis, dialectics and pragmadialectics. Last but not least, 23% of the studies use the analytical framework of argument skills proposed by Kuhn (1991), initially as a tool to analyze participants´ interview answers on everyday
issues, and later adapted to dialogue and interaction analysis (e.g. Kuhn & Felton, 2001). Regarding our first question on whether there is a significant correlation between tool of analysis and year of study, the answer is positive. As shown on Table 1, the use of Toulmin´s model, either in its original form (Toulmin, 1958) or adaptations of it (e.g. Toulmin, Rieke, Janik, & Allan, 1984; Erduran et al., 2004), has increased in the last years, together with a re-awakening of Kuhn´s tool and a steady increase of other informal logic tools (especially interaction analysis, dialectics and pragmadialectics).
We also asked whether there is a significant correlation between tool of analysis and more specific study characteristics, presented in the Method section. More precisely, we did not find any significant correlation [Chi2 (2) = 3.3, ns] between study type and use of specific analytical tool, which means that the fact of conducting an experimental study or not does not influence on the researchers´ decision regarding using one tool or another. In regards to whether the task assigned to the study participants was a production, an assessment, or a combined task, again no significant differences in the distribution of frequencies of tool of analysis were found [Chi2 (4) = 4.8, ns]. However, we observed a tendency in not
considering Kuhn´s tool as appropriate for assessment tasks, but rather for production (oral, written, semi-oral) tasks. As seen on Table 2, for assessment tasks, which are mostly related to the identification of fallacies, dialectical informal logic tools are preferred.
As far as the age of participants is concerned, no significant correlation was found [Chi2 (6) = 1.1, ns], which means that whether the study focuses on children, adolescents, or adults is not a factor influencing on the tool´s decision. On the other hand, when the subject of argumentation is considered, there is a strong tendency (Chi2 (4)=24.6, p<.001) in using Toulmin´s tool and its adaptations for natural sciences, whereas Kuhn´s and other informal logic tools for general issues. Table 3 shows the distribution of frequencies.
When it comes to independent variables and their use in combination with “Toulmin”, “Kuhn” or “Other”, a rather homogeneous distribution appears as shown in Table 4. In spite of the low frequencies, we still may observe a qualitative tendency: When the intervention is at a general level, i.e. explicit argument teaching, Toulmin seems to work, but when it becomes more concrete or combined with other variables, Kuhn and other tools are preferred.
Last but not least, some interesting insights emerged when we crosstabulated the tool of analysis used with some main dependent variables related to argument characteristics assessed by each reviewed study. More precisely, we identified the following three significant correlations, presented in detail on Table 5:
a. For the assessment of the quality and quantity of grounds and reasons, Toulmin is considered more appropriate, followed by Kuhn and then others [Chi2 (2) = 22, p=.001];
b. For the assessment of the quality and quantity of counterarguments and rebuttals, Kuhn is considered more appropriate, followed by Toulmin and then others [Chi2 (2) = 6.2, p=.05];
c. For the assessment of argument strategy, other tools are considered more appropriate, followed by Kuhn and then Toulmin with no studies at all [Chi2 (2) = 13.6, p=.001].
Based on these findings, a taxonomy of preferences towards one of the three most frequent types of argument analysis tools (i.e. Toulmin, Kuhn, and Other) based on the researcher´s focus each time may be proposed, as presented in Table 6.
In this paper, we studied the dominance of argument analysis and assessment tools in regards to their use by educational researchers. Our main motivation lies on the fact that most educational researchers still prefer to use Toulmin´s model (TAP), which is based on the structural rather than strategic aspects of arguments. The main question was whether the choice for TAP or other tools is systematically related to any study characteristics, and, thus, we can identify any trends in educational research in argumentation.
After a careful analysis of both primary and secondary sources, we found that some significant correlations between tool of analysis and study characteristics do exist, especially regarding the year of study and the type of dependent variables involved. The subject of argumentation and the task proposed to the participants are also related to the researchers´ preference for TAP, which is mainly the choice for both production and assessment tasks in the field of natural sciences. On the other hand, when the research focus is more general, the design more complex (variety of independent variables) and the unit of analysis is either argument strategy or argumentative interaction, Toulmin is not the first preference of educational researchers, who also tend to use Kuhn´s analytical categories or other proposals such as argument schemes, dialectics and pragmadialectics. However, TAP continuous to be the most predominant (but not necessarily the better justified) choice for researchers who endeavour to analyse and assess students and teachers argument skills.
This paper led to some first insights regarding what guides educational researchers towards the choice of a tool of argument analysis or another. It was shown that the choice for TAP or other tool is not hazardous, but it is based on criteria related to the study´s focus and characteristics. The taxonomy presented on Table 6 might guide future educational researchers regarding their choice of one tool or another in respect to what others have done thus far.
The pre-dominance of Toulmin´s pattern as an argument analysis tool has been confirmed also in this study. However, this fact alone does not make TAP or any other tool more powerful or more appropriate. Our duty as argumentation scholars is to look at the main challenges of education research in argumentation and try to find ways to better approach those. For example in our previous study (Rapanta et al., 2013) we showed that the main argument skills that teachers-researchers focus on are the correct use of evidence, the distinction of arguments from explanation, and the achievement of learning goals through arguing. More researchers should focus on the profound assessment and enhancement of these skills and others. At the end of the day, our question should not be What comes after Toulmin? but what hinders educational researchers from applying other tools of assessment, more related to the well-known informal logic standards of quality, such as the relevance, sufficiency and acceptability of the arguments and counterarguments used in different contexts of the curriculum.
Barth, E. M., & Krabbe, E. C. (1982). From axiom to dialogue: A philosophical study of logics and argumentation. Berlin & New York: Walter de Gruyter.
Clark, D. B., Sampson, V., Weinberger, A., & Erkens, G. (2007). Analytic frameworks for assessing dialogic argumentation in online learning environments. Educational Psychological Review, 19, 343-374.
De Vries, E., Lund, K. & Baker, M.J. (2002). Computer-mediated epistemic dialogue: Explanation and argumentation as vehicles for understanding scientific notions. The Journal of the Learning Sciences, 11(1).
Duschl, R. A., & Osborne, J. (2002). Supporting and promoting argumentation discourse in science education. Studies in Science Education, 38(1), 39-72. Doi: 10.1080/03057260208560187
Erduran, S., Simon, S., Osborne, J. (2004). TAPping into argumentation: Developments in the application of Toulmin’s Argument Pattern for studying science discourse. Science Education, 88(6), 915-933.
Felton, M. (2004). The development of discourse strategy in adolescent argumentation. Cognitive Development, 19(1), 39- 58.
Felton, M., & Kuhn, D. (2001). The development of argumentative discourse skills. Discourse Processes, 32, 135-153.
Hitchcock, D., & Verheij, B. (2005). The Toulmin model today: Introduction to the special issue on contemporary work using Stephen Edelston Toulmin’s layout of arguments. Argumentation, 19(3), 255-258.
Johnson, R. H. (2000). Manifest rationality: A pragmatic theory of argument. London: Routledge.
Kelly, G. J., Druker, S., & Chen, C. (1998). Students’ reasoning about electricity: combining performance assessments with argumentation analysis. International Journal of Science Education, 20(7), 849-871.
Leitao, S. (2000). The potential of argument in knowledge building. Human Development, 43, 332–360.
Mcalister, S., Ravenscroft, A., & Scanlon, E. (2004). Combining interaction and context design to support collaborative argumentation using a tool for synchronous CMC. Journal of Computer Assisted Learning, 20(3), 194-204.
Nussbaum, M. (2011). Argumentation, dialogue theory, and probability modelling: Alternative frameworks for argumentation research in education. Educational Psychologist, 46(2), 84-106.
Prakken, H. (2005). Coherence and flexibility in dialogue games for argumentation. Journal of logic and computation, 15(6), 1009-1040.
Rapanta, C., Garcia-Mila, M., & Gilabert, S. (2013). What is meant by argumentative competence? An integrative review of methods of analysis and assessment in education. Review of Educational Research, 83(4), 483-520.
Rigotti, E., & Morasso, S. G. (2010). Comparing the Argumentum Model of Topics to other contemporary approaches to argument schemes: the procedural and material components. Argumentation, 24(4), 489-512.
Sampson, V., Clark, D.B. (2009). A comparison of the collaborative scientific argumentation practices of two high and two low performing groups. Research in Science Education, DOI 10.1007/s11165-009-9146-9.
Simon, S., Erduran, S., & Osborne, J. (2006). Learning to teach argumentation: Research and development in the science classroom. International Journal of Science Education, 28(2, 3), 235–260.
Toulmin, S. (1958). The uses of argument. Cambridge: Cambridge University Press.
Toulmin, S. E., Rieke, R. D., Janik, A., & Allan. (1984). An introduction to reasoning. New York: Macmillan.
Van Eemeren, F. H., & Grootendorst, R. (2004). A systematic theory of argumentation: The pragma-dialectical approach. Cambridge, UK: Cambridge University Press.
Voss, J.F. (2005). Toulmin´s model and the solving of ill-structured problems. Argumentation, 19(3), 321-329.
Voss, J. F., & Means, M. L. (1991). Learning to reason via instruction in argumentation. Learning and instruction, 1 (4), 337-350.
Voss, J. F., Perkins, D. N., & Segal, J. W. (Eds.) (1991). Informal reasoning and education. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Walton, D. N. (1996). Argumentation schemes for presumptive reasoning. Mahwah, New Jersey: Lawrence Erlbaum Associates.
Walton, D. N., & Krabbe, E. C. (1995). Commitment in dialogue. Basic concepts of interpersonal reasoning. Albany, New York: State University of New York Press.
Walton, D. N., Reed, C., & Macagno, F. (2008). Argumentation schemes. Cambridge: Cambridge University Press.