ISSA Proceedings 1998 – Critical Thinking: Assessment, Flow Charts And Computers

This paper will look at some new directions in the teaching of critical thinking. This project originally began as an assessment project to discover how well our students were mastering the critical thinking unit in our introductory philosophy course. By using computers to test the pre and post course skills of students, and by running some statistical analyses of what students were and were not learning, I became aware that students had little difficulty memorizing logical concepts – they could define arguments, they understood the difference between premises and conclusions, etc.What they were not able to do successfully, or as successfully as I would like, is apply these concepts to new material. They had difficulty distinguishing arguments from other forms of discourse, evaluating new arguments for strength and validity and recognizing examples of pseudoreasoning. What they most needed help in was learning the skills one uses to come to the decision that a passage does or does not contain an argument, or that a particular form of fallacious reasoning is being used.
My initial computer exercises focused on reinforcing the nature of the concepts – what an argument is, what a slippery slope involves, distinguishing between valid and invalid arguments, etc. These exercises improved student outcomes, but not as significantly as I had hoped. My next step was to develop flow charts to help students picture graphically the relevant reasoning processes. I have used three such charts, designed to help students recognize arguments, recognize valid arguments, and recognize several informal fallacies. The students could then use these flow charts to develop their own methods to accomplish these tasks.
By focusing on the processes used to make logical decisions, I hope to show that students can master logical concepts more easily. Most logic texts are problem based; yet little is offered on processes to solve the problems. For example, most texts include problems on identifying arguments, but do not show the steps necessary to distinguish arguments from other types of discourse. Notable exceptions to this are units on more complicated logical procedures such as diagraming arguments, using Venn Diagrams and logical proofs. Logic Texts address part of this problem when they teach students how to recognize premises and conclusions. The expectation seems to be that if students can understand the concept of an argument, they can therefore identify arguments in practice. But I do not find this to be the case. This is not enough to give students the ability to distinguish arguments from other types of discourse.

To address the problem of making the process more explicit, this semester I measured the growth in critical thinking ability of 150 students in three sections of introductory philosophy. The first step was a pre-test on the second day of class to evaluate their ability to recognize arguments, to judge good arguments, and to detect examples of informal fallacies. The test consists of 14 computer questions that ask students to distinguish arguments from other forms of discourse and to say whether the conclusion of an argument follows from the premises given. It also includes 11 questions given in class that asks what is wrong with passages that each contain an example of an informal fallacy. The same test was repeated on the last day of class. The students never received the results of their pre-assessment test or discussed the correct answers in class.
Because the concept of flow charts is integral to the way computers are programmed, they can be programmed to duplicate the kinds of flow charts employed here. By using pre and post course tests for assessment purposes, I hoped to show that through using flow charts, and computer exercises based on those flow charts, students’ acquisition of these critical thinking skills would increase.

The first step in this process was the decision about which skills to target in my course. Since philosophy is the discipline that employs argumentation most prominently, and since this course fulfills the University requirement for critical thinking, students must learn to identify arguments and to distinguish arguments from other forms of discourse. This pushed me immediately back into the arena of concepts, but there did not seem to be much literature on the process of recognizing arguments. From treatments of the nature of arguments: I chose the definition used by Moore and Parker in their text Critical Thinking, one of the most popular texts for critical thinking courses in the United States. They begin their exploration of arguments with the claim that an argument is an attempt to settle an issue (something up for debate) through the use of premises and conclusions supported by premises (Moore and Parker 97: 8-11). This seemed a promising avenue for exploration,
though it proved to generate some difficulties as well. Using this definition, I instructed students to look for an issue, a conclusion and some support for the conclusion.
However, this definition led substantial numbers of students to deny that the following is an argument: All men are mortal; Socrates is a man; therefore Socrates is mortal. They rejected this as an argument on the grounds that there is nothing to dispute, or no issue. I therefore revised Moore and Parker’s criteria for an argument to the following: an argument must involve an attempt to persuade, must come to a definite conclusion and must provide reasons to accept that conclusion. Students find these criteria somewhat easier to follow than Moore and Parker, though they still have some difficulty in deciding whether a passage involves an attempt to persuade.

The flow chart I developed for students to use is #1 on the handout. It works reasonably well: scores on the homework and quizzes for this section of critical thinking have improved dramatically. This area of the assessment had been one that, before the use of flow charts, showed very little improvement from the beginning of the course to the end. In my original assessment the average improvement on this section was less than 5%; using the flow charts this semester the average improvement more than doubled to over 11.29%. I am not completely satisfied with the current flow chart (perhaps more needs to be said about what constitutes an attempt to persuade, and it does not address some of the subtle differences between explanations, justifications and arguments). Still, it seems to help students to improve their ability to recognize arguments.
The second skill I chose to address as part of critical thinking was the evaluation of arguments: specifically, the ability to distinguish valid deductive arguments from invalid ones, and the ability to decide an argument’s soundness Good arguments are important precisely because we can trust their conclusions. So it is essential for critical thinkers to be able to distinguish good arguments from bad. The conclusions of sound arguments are, by definition, true; so the ability to pick out such arguments is an essential skill.

Since the list of valid arguments is so extensive, and given the time constraints in an introductory course, I decided to choose just a few for this unit. Arguments that use hypothetical seemed a good start, because students initially find these difficult, and because they are a source of many reasoning mistakes. To illustrate such mistakes I usually let students read some examples of valid and invalid modus ponens and modus tollens and have them make intuitive suggestions about the reasoning in each. Invariably, they argue that the valid forms are bad arguments and the invalid forms are good arguments. Despite this poor beginning, grades on homework for this unit after the introduction of the flow charts are the highest in the course.
Besides hypothetical arguments, we also look at the validity of disjunctive syllogisms, another source of reasoning mistakes commonly made by students. Most students understand the word “or” only in its exclusive sense, meaning only one alternative is the case. So they commonly reason that if A is true, B cannot be true. With some exposure to the inclusive sense of “or,” most students are able to avoid this reasoning mistake, though for some students disjunctive arguments are the hardest to evaluate and they continue to regard all “or’s” as exclusive. ( See Flow Chart #2) Test scores and homework scores on the evaluation of arguments show considerable improvement with the use of flow charts.
But the most dramatic improvement on the assessment test came in the section on informal fallacies, despite the fact that I am the least satisfied with the flow chart I developed for this purpose. Since informal fallacies are so widespread in everyday life, from the comics section of the newspaper to political oratory to advertising, all of us are bombarded with examples of informal fallacies. This made me conclude that the ability to recognize such fallacies and to understand why they are compelling for many people is an extremely important skill for a critical thinker. Developing a flow chart to duplicate these processes proved the hardest challenge.
Over the years, in teaching such reasoning mistakes, I have encountered resistance from students who find these concepts vague and difficult to master. The task was made more difficult by the fact that no two logic texts approach informal fallacies in the same way, or even agree on a list of such fallacies. The most helpful text here was Morris Engel’s With Good Reason, because of the way he classifies the mistakes(Engel 94: 84-86). I also found the treatment of informal fallacies in Cederblom and Paulsen’s text, Critical Reasoning helpful in coming up with a procedure for identifying such fallacies (Cederbloom and Paulsen 91: 134-166).

I tell my students that most informal fallacies use five kinds of appeals in their proofs: diversion, emotion, presumption, misuse of language, and appeals to the presenter of an argument. If they can identify what the author is attempting to use for proof, they can usually correctly identify such fallacies as ad hominem, ad populam, etc. Some of the categories are easier to recognize than others: appeals to emotion are much easier to identify, for example, than what Engle calls fallacies of presumption. This leads me to suggest a process of elimination as a part of the flow chart for this unit. (See Flow Chart #3) One of the chief difficulties in constructing flow charts for these kinds of exercises is that more than one fallacy can be involved, depending on the interpretation of the passage. Refinement in the charts may needed to provide branches that reflect the overlaps among the fallacies. Still, though there is room for improvement, students increased their mastery of these concepts by an average of 154% since they began using the charts.
I had hoped to translate this approach into a set of computer questions that duplicate the flow charts. I have written the basic outlines for such a project, even written the preliminary exercises. My current computer exercises are written in tree form with students answering relevant questions and then being given explanations of those answers. The software that is used for those exercises is Authorware by Macromedia and it will be no major project to rewrite these so that the questions duplicate the questions on the flow chart. Unfortunately our Department’s computer expert got more interested in protesting Texas’ marijuana laws than in improving critical thinking. As result of his public pot smoking (perhaps in itself a lapse of critical thinking), he was arrested and expelled from the University. Consequently, the exercises I had planned to be performed on the computer were never programmed into the machines.
I believe that using such exercises will continue the improvement already achieved by the flow charts. Overall, my students demonstrated more than a 100% average improvement in their scores on the post-assessment test; this compares with a 46% average improvement in scores using the computers but without the flow charts. The average score on the department-wide assessment also increased from 4.84 out of 10 to 7.14: a 47% increase. This compares with an average 25% increase before using the flow charts.

My basic contention, then, is that in teaching logic and argumentation, we must focus more on the processes we use in good argumentation and reasoning rather than the concepts. Students seem to understand the definition of premise and a conclusion, but frequently can not distinguish them in actual arguments they encounter in real life or even in logic books. Logic texts have always focused on the doing of logic through the use of exercises that emphasize skills. What I found missing and what my students profit from is more explanation of the very basic processes involved in mastering those skills.
Ironically, those of us who teach logic or critical thinking may be the least able to explain these processes. They have become so automatic for us that we rarely stop to think about the steps we go through to recognize arguments, evaluate them, or pick out instances of informal fallacies. We understand the concepts on an abstract and even on a practical level, but we rarely stop to go through the processes and make them fully explicit for our students.

This became clearly apparent to me as I tried to develop flow charts for my students. It was very difficult for me to say why I thought something was or was not an argument. And I frequently found my self disagreeing with the authors of a particular text. The following appears in Moore and Parker’s supplement to Critical Thinking: The Logical Accessory. “Some of these guys that do Elvis Presley imitations actually pay more for their outfits than Elvis paid for his. Anybody who would spend thousands just so he can spend a few minutes not fooling anybody into thinking he’s Elvis is nuts” (Moore and Parker 95:33). Moore and Parker do not feel this is an argument, and some of the time I agree with them that neither sentence really supports the other. But other days I can see my students’ point that there does seem to be an attempt to persuade; there does seem to be a definite stand, and some reason is given for that stand. Perhaps we have not yet gotten to the heart of the concepts. Perhaps if we more fully understood the nature of logical concepts, the processes would not be so difficult to explain to our students. I don’t really want to push that line of thought, so much as to suggest that we need to spend more time discovering the processes that lie behind logical thought and reasoning. My flow charts are an initial attempt to explore this area; they begin to meet what students seem to need. They help them to understand how we make decisions that something is or is not an argument, is a good argument or is an example of an informal fallacy. I would very much welcome any suggestions that you might make as to a better analysis of the processes involved.

APPENDICES
#1 Flow Chart for recognizing arguments
1. What is this passage trying to do?
Present facts – no argument
Describe something – no argument
Present compound unrelated claims – no argument
Persuade me about the truth of a claim – possible argument –Procede further.

2. What is the claim or issue at stake?
State this in your own terms. Go to step 3

3. Does the passage take a clear stand on the isue? What is the stand?
If no stand, no argument.
If yes, procede to step 4.

4. Does the passage provide clear reasons to accept the stand taken?
If no reasons, no argument.
If yes – then argument.
=
An argument must be an attempt to persuade, include a definite stand on an issue and provide reasons to accept that stand.

#2 Flow Chart for Evaluating Arguments
Find the logical indicator – If move to step 2; if there is more than one if statement move to #9 If the logical indicator is an or move to #13.

2. Label the claims beginning with the if clause, no matter what comes first in the argument. Label the antecedent or if clause p; label the consequent or then clause q.

3. Identify the conclusion; label the claim based on the first premise.

4. Identify the second premise(this will be what is left over). Label the claims according to the first premise.

5. Set up the schema.

6. Identify the argument using the schema: if the second p and q are affirmations or repeat the first p and q, the argument is a modus ponens. Go to #7 If the second p and q are denials the argument is a modus tollens: go to #8.

7. Determine validity: A modus ponens must affirm the antecedent clause (p clause) to be valid. If it affirms the consequent clause (q clause) it is invalid.

8. Determine validity of modus tollens: A modus tollens must deny the consequent clause (q clause) to be valid. If it denies the antecedent clause (p clause) it is invalid.

9. More than one if statement means the argument is a chain or hypothetical syllogism.

10. Find the conclusion. Label the premises with p, q and r first. Label the conclusion last.

11. Set up the schema and determine validity. Watch for breaking the chain or reversing the conclusion. To be valid the second premise should affirm the consequent clause of the first premise in the second premise and should include the antecedent clause of the first premise and the consequent clause of the last premise in the conclusion. Any other pattern is invalid.

12. If the logical indicator is an or , first determine whether it is a strong or weak disjunct. ( In a strong disjunct only one alternative is possible.)

13. Label the claims beginning with the or statement. Set up the schema with the conclusion last. Determine validity: all strong disjunct are valid; in a weak disjunct only the denial in the premisses is valid; if the denial is in the conclusion it is invalid.

#3 Flow chart for recognizing informal fallacies
What is the main claim or the conclusion?
What are the premises or support?

Do the premisses or conclusion contain a word or phrase that could have more than one meaning?
one tipoff – a word used more than once.
NO – Move to next question
YES – 1. Ambiguous word or phrase – EQUIVOCATION
2. Sentence structure is ambiguous==AMPHIBOLY
3. Grouping is ambiguous-moves from parts to whole ==COMPOSITION
4. Grouping is ambiguous – moves from whole to parts==DIVISION

Are the premises irrelevant to the main claim?
NO – Move to the next Question
YES – 1. Changes subject==SMOKESCREEN
2. Appeals to others opinions==APPEAL to BELIEF/COMMON PRACTICE ( See also emotions)
3. Appeals to undesirable consequences==SLIPPERY SLOPE
4. premises distort main claim==STRAW MAN

Do the premises appeal to emotions or supply motives?
NO – Move to next question.ES
YES – 1. They appeal to the good opinions of others== PEER PRESSURE
2. They appeal to wealth and status==-SNOB APPEAL
3. They use flattery== APPLEPOLISHING
4. They use fear== SCARE TACTICS
5. They appeal to our sense of compassion==PITY
6. OTHER EMOTIONS, eg. Spite, ridicule, etc

Do they attack the presenter of the argument or use the presenter’s status illegitimately?
NO – move to the next question
YES – 1. Attacks the person directly==AD HOMINEM (abusive)
2. Attacks person or claim because of source==AD HOMINEM (genetic)
3. Discredits source for inconsistency==AD HOMINEM (inconsistent)
4. Uses a source outside its field of expertise== AD VERECUNDIAM Also called illegitimate authority

Is there an unjustified assumption? This category is usually reached by elimination. If none of the other categories fit try one of the following:
1. Look for premises and conclusion that say the same thing in slightly different terms==BEGGING THE QUESTION
2. Look for unproven assumption that there are only 2 alternatives.(MAY BE STATED AS AN IF CLAUSE)=FALSE DILEMMA
3. Look for claim that lack of proof proves the other side==ARGUMENT FROM IGNORANCE
4. Look for improper relationships between causes and effects ==FALSE CAUSE
5. Look for conclusions based on too little evidence or illegitimate evidence==HASTY GENERALIZATION
6. Look for a claim that assumes that an earlier question has already been answered in a particular way==COMPLEX QUESTION

REFERENCES
Cederblom, Jerry and David Paulsen (1991), Critical Reasoning. Belmont, California: Wadsworth Publishing Company.
Engle, Morris (1994), An Introduction to Informal Fallacies. New York: St. Martin’s Press.
Moore, Brooke Noel and Richard Parker(1997), Critical Thinking. Mountain View, California: Mayfield Publishing Company.
Moore, Brooke Noel and Richard Parker(1995), The Logical Accessory: Instructor’s Manual to Accompany Critical Thinking. Mountain View, California: Mayfield Publishing Company.