ISSA Proceedings 2014 – Conductive Argumentation, Degrees Of Confidence, And The Communication Of Uncertainty

Abstract: The paper argues that there is an epistemic obligation to communicate the appropriate degree of confidence when asserting conclusions in conductive argumentation. Contrary to the position of some theorists, we argue that such conclusions frequently are, and should be expressed with appropriate qualifications. As an illustration, we discuss the case of the Italian scientists tried for failing to convey to the public appropriate warnings of the risks of the earthquake in L’Aquila.

Keywords: conductive argumentation, judgment confidence, expression of uncertainty

1. Prologue
On April 6, 2009, a magnitude 6.3 earthquake struck L’Aquila, Abruzzo, resulting in considerable devastation and the death of 300 people. Seven Italian officials and scientists were subsequently put on trial for manslaughter. The accusation was that scientists presented incomplete, inconsistent information which falsely assured the public and caused the deaths of 30 residents. The usual practice when an earthquake was likely was for residents to sleep outside, but it was alleged that because of the assurance, these individuals remained in their houses and were killed in the quake (Ashcroft 2012). The prosecution argued that the assessment of risk communicated to the public was unjustifiably optimistic and that lives could have been saved had people not been persuaded by the assurances to remain in their houses (Hooper 2012). In 2012, the scientists were found guilty of manslaughter and sentenced to six years in prison.

We will return to this case later. We have no intention to try to evaluate its merits, but we shall examine the issues it raises regarding the obligation to communicate an appropriate degree of certainty or uncertainty in one’s judgments.

2. Introduction
This paper begins by making the argument that a degree of uncertainty is an unavoidable aspect of conductive argumentation. The arguments which comprise instances of conductive argumentation vary in terms of the degree of support that they provide for their conclusions; for this reason the strength of the judgments warranted by particular instances of conductive argumentation will vary as well. We argue, further, that this variability imposes an epistemic requirement on arguers to apportion the confidence of their judgment to the strength of the reasons. Moreover, because of the dialectical nature of argumentation, there is the additional requirement for arguers to communicate the appropriate degree of certainty or uncertainty when making judgments in the context of an argumentative exchange.

3. Argumentation and uncertainty
The traditional focus for the philosophical study of argumentation has been individual arguments, in terms of both their structure and their evaluation. The model of argument which has been dominant has been deductive argument, i.e., an argument whose premises entail the conclusion. Provided that the premises are true, the conclusion follows with certainty. Uncertainty may, of course, still arise with respect to the truth of the premises.

This requirement of inference certainty does not, however, fit a great deal of actual argumentation, as has been pointed out by theorists since the inception of the Informal Logic movement. In probable reasoning, for example, the conclusion does not follow necessarily but only with some degree of probability (Blair & Johnson 1987, p. 42). The situation is similar for inductive reasoning: “Inductive inferences vary from weak to strong; there is no all-or-nothing critique such as ‘valid-or invalid’ available” (Blair & Johnson 1987, p. 42).

Theorists have, however, been increasingly broadening their focus from exclusively individual arguments to the entire enterprise of argumentation. Argumentation can be conceptualized as a socio-cultural activity (Hitchcock 2002, p. 291) which is dialectical in the sense that it involves an interaction between the arguers and between the arguments (Blair & Johnson 1987). This focus is much broader than the making of individual arguments. Rather, arguments are put forward, criticisms and objections offered, responses proposed, and, frequently, revisions made to initial positions (Bailin & Battersby 2009). It is this practice of argumentation that is our focus here, and in particular the practice of conductive argumentation (or conductive reasoning). By conductive reasoning we are referring to the process of comparative evaluation of a variety of contending positions and arguments with the goal of reaching a reasoned judgment on an issue (Battersby & Bailin 2011). Such judgments are generally based on the weighing of both pro and con considerations.

The focus of many theorists working in the area is, however, on individual conductive arguments rather than on conductive reasoning. Conductive arguments are, as Govier puts it, “arguments in which premises are put forward as separately and non-conclusively relevant to support a conclusion, against which negatively relevant considerations may also be acknowledged” (Govier 2011, p. 262). In our view, however, viewing conductive reasoning in terms of individual arguments fails to due justice to the dialectical nature of argumentation (Battersby & Bailin 2011). In addition, attempting to make conductive reasoning fit into the traditional model of argument structure has resulted in unnecessary conundrums, for example how to analyze counter-considerations (are they premises? counter-premises?) or how to diagram these anomalous types of arguments. Our focus, in contrast, is on conductive reasoning more broadly. According to this perspective, the structure of conductive argumentation is viewed in terms of a balancing of competing arguments and claims rather than as a single argument.

4. Uncertainty in conductive argumentation
There are a number of reasons why conductive argumentation does not lead to conclusions which can be asserted with epistemic certainty. These include inferential uncertainty, the inherent uncertainty of particular claims and judgments, the open-endedness of the reason-giving process, and variability in the weighing of pro and con considerations. Because of these factors, the degree of certainty with which conclusions of conductive argumentation can justifiably be held will vary.

Inferential uncertainty is a feature of conductive reasoning just as it is with inductive reasoning. Given that particular claims are true, there is still the question of how much support they give to the conclusion.

The uncertainty has also to do with the inherent uncertainty of particular claims and judgments which go into the reasoning process. The likelihood of factual claims is an important factor in evaluating their weight as the greater the likelihood of the claim, the more weight it can add to the conclusion. Likelihood is, however, often difficult to determine. To compound the difficulty, any argument leading to a judgment about what to do must also take into account future states of affairs which are usually even less certain than judgments about current states of affairs. What one can do in both these cases is to use the available information, history, contextual factors, and statistical tools to make reasoned judgments. And in the area of moral issues, while there are some widely accepted general moral principles, their application in particular cases inevitably creates some degree of uncertainty, the degree depending on the strength of the supporting arguments (Battersby & Bailin 2011).

The uncertainty arises also from the nature of conductive reasoning itself. One important factor is the open-endedness of the reason-giving process. Competent conductive reasoning requires laying out the dialectic – the arguments on various sides of the debate, as well as objections to the arguments and responses to the objections. No survey of arguments will be exhaustive, however. The possibility always exists that additional reasons and arguments will be put forward which might affect the outcome of the reasoning (Battersby & Bailin 2011). This being said, the more extensive the review of the available evidence and argumentation, the stronger the support for the resultant judgment.

Uncertainty also comes in due to the process of weighing the various reasons pro and con. There is sometimes variability amongst arguers in the evaluation of the comparative strength of evidence and arguments on different sides of an issue and disagreement about the appropriate weight to be apportioned to various considerations. This is not to say that weightings are (primarily) subjective. Weightings can be justified (or criticized) by appeal to objective factors and considerations (e.g., the likelihood of claims, appeal to widely shared values and principles,). Nonetheless, there may not be consensus on how some considerations should be weighted and there may be more than one judgment which is defensible given the context (Battersby & Bailin 2011).

Because of the uncertainty of particular claims, the variability in the evaluation of the comparative strength of evidence and arguments, the different weightings given to various considerations, and the open-endedness of the reason-giving process, an instance of conductive reasoning can, at best, offer good reasons and strong support for a conclusion but not certainty.

This does not mean, however, that it is not possible to make warranted judgments in instances of conductive reasoning. Guidelines exist for making reasoned judgments and criteria exist for their evaluation (Battersby & Bailin 2011). What it does mean is that there will always be some uncertainty with respect to the judgments emerging from the process of conductive argumentation and that the strength of the judgments warranted by particular instances of conductive argumentation will vary.

5. Confidence in judgment
The strength of the evidence and argumentation in support of conclusions in conductive argumentation will vary from case to case (Battersby & Bailin 2011). In some cases the evidence for a particular judgment may be overwhelming. There are, for example, very strong reasons to believe that smoking causes cancer or that the enslavement of human beings is morally unjustifiable. In other cases the weight of reasons may favour a particular judgment but not without significant opposing reasons or counter considerations. Claims about the causes of climate change might fall into this category. In still other cases, the reasons may be insufficient for reaching a judgment, for example in debates about life on other planets. Thus, in robust argumentation, warrant is usually a matter of degree.

Engaging in the process of argumentation imposes certain epistemic requirements on arguers: that they present arguments justified by the available evidence, address appropriate objections and provide reasonable responses, and revise their initial position when warranted. But the variability in the degree of support for different judgments also imposes an additional requirement on arguers: that they apportion the confidence of their judgment to the strength of the reasons. Not all judgments warrant an equal level of confidence. It is important to be clear that we are not referring to subjective confidence – how confident an individual may happen to feel about a judgment, but rather rational or warranted confidence – the level of confidence that is justified by the reasons and evidence.

The following is a schema which we have developed to represent the level of confidence warranted by different weights of reasons:

• A very confident judgment is warranted when the weight of reasons clearly supports the judgment.
• A reasonably confident judgment is warranted when the weight of reasons strongly supports the judgment but there are still strong countervailing considerations.
• A tentative judgment is warranted when the weight of reasons is not overwhelming but is supportive of one position, and we can make a judgment on balance.
• A suspended judgment is warranted when the reasons for different positions are closely balanced or when there is insufficient evidence to make a judgment.
This schema has similarities to the categorization used for classifying the strength of causal inferences in science (US Department of Health, 2006).

These four levels of judgment confidence are not discrete but can be seen as marking positions along a continuum. The categorization allows for a range of possibilities in between.

Apportioning one’s confidence in a judgment to the strength of the reasons is always epistemologically significant. It is when there is a need to act on the basis of our judgments, however, that the issue of how justified our confidence is in our judgments becomes crucial. The greater the consequences of action (or inaction), the greater the need for a level of argumentative support that warrants a confident judgment. A useful comparison can be made to legal judgments. In criminal cases, where there is a great deal at stake (freedom versus imprisonment, or even life versus death), the standard of proof is beyond a reasonable doubt, which requires a level of evidence sufficient to warrant a very confident judgment. In civil matters, where there is usually less at stake, the standard of proof is usually balance of probabilities, which clearly requires only an on balance judgment.

6. Degrees of certainty or uncertainty
The fact that argumentation is dialectical imposes yet a further requirement on arguers. It is not just a matter of apportioning one’s confidence in a judgment to the strength of the reasons. There is also a requirement to communicate the appropriate degree of certainty or uncertainty when making judgments in the context of an argumentative exchange.

There are many ways in which one’s confidence in a judgment and hence the degree of certainty or uncertainty may be expressed:

• A very confident judgment implies a high level of certainty and would be marked linguistically by such phrases as “I am very confident that,” “it is clear that,” “there’s little doubt that,” “the evidence strongly indicates that.”
• A reasonably confident judgment implies a moderately high level of certainty and might be indicated by such phrases as “I am reasonably sure that,” “it seems very likely that,” “the evidence by and large indicates that.”
• A tentative judgment implies some degree of uncertainty, although not enough to preclude making a judgment. A tentative judgment may be indicated by such phrases as “it appears on balance that,” “the weight of evidence tips somewhat in favour of,” “my tentative conclusion is that.”
• A suspended judgment implies a high level of uncertainty and would be indicated by such phrases as “there is not enough evidence to make a judgment,” “the reasons on both sides seem equally balanced,” “the judgment will have to be deferred until more evidence is available,” “the jury’s still out on this.”

7. An objection
Curiously some theorists have denied that conductive arguments can have a conclusion that expresses uncertainty. In a recent posthumous publication, Adler argues against the claim that countervailing considerations detract from the support for the conclusion in a conductive argument:

The claim that I dispute is that once the conclusion is drawn, the counter- considerations continue to diminish its support (Adler 2013, p. 4).

As a consequence:

… the conclusion of a Conductive Argument is characteristically detached and accepted without (epistemic) qualification (Adler 2013, p. 6).

And further:

Let me summarize my reasons for taking Conductive Argument to characteristically lead to unqualified conclusions that are accepted and asserted (Adler 2013, p. 6).

If we understand him correctly, he is arguing that if we are asking an interlocutor to accept our conclusion, then we are always asking him to accept the conclusion without the modifiers of “all things considered,” “on balance,” “it is very likely that” etc.

It is significant that Adler’s objection is framed in terms of conductive arguments while we frame the issue in terms of conductive argumentation. The difference in framing is important in terms of the consideration of his objection, a point to which we shall return.

We would maintain that qualified conclusions are common in conductive argumentation. In arguments for factual claims, expressing uncertainty is not unusual, e.g., “The forecast notwithstanding, it looks like it might rain.” “Even though he doesn’t like parties, Tom is a good friend so he’ll likely come to my birthday party.” “There are many fine contemporary authors, but she is probably the best of her generation.” The communication of the degree of certainty of findings is also a common practice in the kind of argument to the best explanation exhibited in scientific reasoning and scientific reports. The following excerpt from an IPCC assessment report on climate change explains the confidence levels used in the report:

The degree of certainty in key findings in this assessment is based on the author teams’ evaluations of underlying scientific understanding and is expressed as a qualitative level of confidence (from very low to very high) and, when possible, probabilistically with a quantified likelihood (from exceptionally unlikely to virtually certain). Confidence in the validity of a finding is based on the type, amount, quality, and consistency of evidence (e.g., data, mechanistic understanding, theory, models, expert judgment) and the degree of agreement. SPM-2

The following examples from the report illustrate the use of these confidence levels:

(1) It is virtually certain that globally the troposphere has warmed since the mid-20th century. More complete observations allow greater confidence in estimates of tropospheric temperature changes in the extratropical Northern Hemisphere than elsewhere. There is medium confidence in the rate of warming and its vertical structure in the Northern Hemisphere extra-tropical troposphere and low confidence elsewhere. {2.4} PSM-4

(2) It is likely that anthropogenic influences have affected the global water cycle since 1960. Anthropogenic influences have contributed to observed increases in atmospheric moisture content in the atmosphere (medium confidence), to global-scale changes in precipitation patterns over land (medium confidence), to intensification of heavy precipitation over land regions where data are sufficient (medium confidence), and to changes in surface and sub- surface ocean salinity (very likely). {2.5, 2.6, 3.3, 7.6, 10.3, 10.4} SPM-13

Although Adler’s argument seems to be directed toward conductive arguments in general (“the conclusion of a Conductive Argument is characteristically detached …”), many of his examples involve practical reasoning, where the conclusion is a decision or recommendation about whether to act. Apparently, he would reject a conclusion that “we should probably do X.” Yet, in practice, we do often qualify a recommendation by “we should probably,” “on balance the best thing to do seems to be,” “there are good reasons to” etc.

Given the frequency of qualified conclusions in conductive argumentation, one might wonder what Adler’s reasons are for denying their possibility. The basis of his argument is a logical one – that in order for a conductive argument to be cogent, i.e., in order for its conclusion to be correctly accepted as true, the conclusion must stand on its own.[i] His focus is on cogent arguments, that is arguments that end inquiry. The alternative for Adler is not qualified conclusions but rather suspended judgment.

It is here that the problem of viewing conductive argumentation in terms of individual arguments becomes manifest. Adler’s analysis has some plausibility when applied to examples such as the classic argument offered by Wellman: Although your lawn needs cutting, you ought to take your son to the movies because the picture is ideal for children and will be gone by tomorrow (Wellman 1971, p. 67). Most of the examples offered by Adler, however, (e.g., mandated health care insurance, stricter rules to restrict immigration, building nuclear power plants) are instances of complex, dialectical argumentation. (Indeed, the distinction between conductive arguments and conductive argumentation is one that Adler himself appears, in places, to acknowledge: Adler, p. 2, footnote 1). In such cases, it is inappropriate to expect certainty (for all the reasons outlined above). It is inappropriate to expect conclusions that are “true”. What we can expect, instead, are judgments that have varying degrees of support.

Adler’s argument does have some prima facie plausibility in that for practical arguments, either we should act, we should not act, or we simply do not know what to do. Indeed, it does seem that when we decide to do something, we have “detached” the decision from the reasoning through our commitment to action. But the detachment is in effect a pragmatic detachment which does not necessarily indicate unqualified confidence, nor will it necessarily end inquiry. On fairly straightforward practical issues, for example which camera to buy, making a decision will likely mark the end of the inquiry. But this may simply be because the action is a fait accompli and does not necessarily indicate a high level of confidence that we have made the right choice. With more complex issues, however, even once an action has been taken, inquiry does not necessarily end, e.g., the U.S. government has made a decision with respect to mandated health care insurance, but the debate has certainly not ended.

It seems to be Adler’s view that it is only detached, unqualified conclusions that “discern or advance and settle new or interesting or important truths, that are worth believing for ourselves or for our audience. They increase our information and expand our corpus of beliefs” (Adler 2013, p. 6). We would argue, on the contrary, that it is appropriately qualified conclusions that really add to our justified beliefs. We are justified in holding our beliefs on such issues with varying degree of confidence commensurate with the strength of the support. Jane’s belief that there should be government mandated health care insurance is one she may hold with considerable confidence given the strength of the reasons in favor and the weakness of the reasons against. She may hold the belief that we should not build nuclear power plants with considerably less confidence given the force of the reasons for as well as against. Adler seems to hold that only unqualified conclusions put “arguers and inquirers in a position that is appropriate to guide further judgments and action” (Adler 2013, p. 6). We would argue, on the contrary, that appropriately qualified conclusions are, in fact, more reasonable guides to action. The conclusions of conductive argumentation are judgments and it is a requirement of reasonableness that such judgments should reflect the degree of support provided by our reasons.

8. Communicating confidence and certainty
We have been arguing, then, that there is a requirement to apportion one’s confidence in a judgment to the strength of the reasons in support of the judgment. We would argue, further, there is also an epistemic and moral responsibility to communicate the appropriate degree of certainty or uncertainty when making judgments in the context of an argumentative exchange. This responsibility arises from the dialectical and interactive nature of conductive argumentation. According to Johnson, that an exchange is dialectical means that “as a result of the intervention of the Other, one’s own logos (discourse, reasoning, or thinking) has the potential of being affected in some way” (Johnson 2000, p. 161). In other words, the reasoning and judgments made by others can and often should affect my reasoning and judgments and form part of the basis for my actions. Just as offering well justified judgments in the context of an argumentative exchange can contribute to others holding better justified beliefs and undertaking better justified actions, so also can communicating one’s judgments at the appropriate level of confidence. Acknowledging uncertainty or confidence as part of one’s judgment or decision to act can inform others of how much confidence you or they should have in the judgment. Communicating a judgment at an inappropriate level of confidence, for example with more confidence than is warranted by the evidence, may contribute to other interlocutors holding beliefs or acting in ways that are poorly grounded.

This responsibility is especially significant when one is in a position of epistemic authority. Experts have an obligation to provide reasons for their judgments, however in contexts requiring expertise, recipients of the judgment are often not in a position to assess the reasoning in any detail. These judgments are generally accepted largely on the basis of trust in the expertise and reliability of the authority. Thus the level of confidence that is expressed in the judgment is an important aspect of the information communicated in the judgment. Returning to the IPCC report, it would be have been misleading if the report had omitted the confidence levels in their various finding. This is especially important as such judgments often form the basis for decisions regarding action, or may themselves be recommendations for action. Compare the following judgments by a physician: (1.) “I have carefully evaluated all the evidence and would not recommend surgery. It is my judgment that it would not help.” (2.) “I have carefully evaluated all the evidence and would not recommend surgery. It is my judgment that surgery is very unlikely to help and the surgical procedure is very risky. But I cannot be 100% confident because there have been a few similar cases where it appears that a surgical invention may have helped to prolong life.” To offer the same conclusion without an indication of the confidence level would be a misleading way of putting forth one’s conclusion. In cases where the argument leads to a somewhat uncertain conclusion based on a balancing of conflicting considerations, failure to indicate the presence of these considerations is an epistemic failure. Given that the purpose of conductive argumentation is to consider countervailing considerations and yet come to a reasonable conclusion, failure to communicate the degree of justification or certainty that the arguments provide also violates basic norms of communication.

9. The l’Aquila case
The trial of the Italian scientists and officials in the L’Aquila earthquake case is a pertinent one to examine with respect to the issue of the communication of certainty or uncertainty. The earthquake had been preceded by a swarm of small quakes, and the charge against the defendants was that they did not do their duty in communicating the likelihood of a major earthquake to the citizens of L’Aquila.

One of the scientists tried, Enzo Boschi, the then-president of Italy’s National Institute of Geophysics and Volcanology, is said to have compared the situation to a large quake that struck L’Aquila in 1703. Boschi is alleged to have said at a meeting in L’Aquila on March 31, 2009, “It is unlikely that an earthquake like the one in 1703 could occur in the short term, but the possibility cannot be totally excluded.” In a press conference after the meeting, Department of Civil Protection official Bernardo De Bernardinis, also a defendant, is quoted (and on video record) as saying that the situation was normal given the context, posing “no danger,” and urging residents to relax (Pappas 2012).

The details of the case are complex and include allegations of political pressure, and of misrepresentation of material. We have no intention to try to evaluate the merits of the case, nor are we in a position to do so. Nonetheless some of the issues raised are pertinent to our discussion. The statements of both Boschi and De Bernardinis would have been grounded in the knowledge that earthquake swarms are very common in seismically active regions such as Abruzzo but only a very small percentage are precursors to major quakes. In fact, seismologists claim that it is virtually impossible to predict major earthquakes. Yet we can note a difference in the level of certainty communicated in the two judgments. Boschi’s judgment that a major earthquake was unlikely could be characterized as a reasonably confident judgment, but in alluding to the possibility of such a quake, it communicated a degree of uncertainty in the judgment. De Bernardinis, in contrast, seemed to be making a very confident judgment that there was no danger of a major quake. His judgment made no reference to the possibility, slight though it may have been. The risk was indeed very low, but not non-existent. Thus his pronouncement, communicated to the public, that there was “no danger” was epistemically overly confident, expressing an unreasonable degree of certainty.

The scientists and officials in question were considered epistemic authorities and the level of certainty communicated by them to members of the public appears to have affected the public’s actions. A local investigator, Inspector Lorenzo Cavallo, is quoted as saying: “The Commission calmed the local population down following a number of earth tremors. After the quake, we heard people’s accounts and they told us they changed their behaviour following the advice of the commission” (Watt, S. 2011). This account is corroborated repeatedly by witnesses testifying at the trial (Billi 2013).

The specifics of this particular case are complex and contested, and it would be inappropriate and imprudent to attempt to pass any judgments. One thing that we do think that the case demonstrates, however, is a strong recognition of the responsibility to communicate the epistemically appropriate degree of certainty or uncertainty in our judgments. It is unreasonable, (epistemically inappropriate) to make or hold a judgment without the appropriate degree of uncertainty given the evidence. It is, in addition, a communicative and perhaps a moral failure to communicate a judgment without the appropriate expression of epistemic uncertainty.

Acknowledgements
We would like to thank Monica Bhattacharjee for her contribution to the preparation of this paper.

NOTE
i. Surprisingly given his thesis, Adler does acknowledge that “there are loads of arguments that end with qualified conclusions, including, ‘plausible’ or, more equivocally, ‘the best explanation is’” (p. 7). But the rest of his argumentation leads us to believe that he would reconcile this apparent contradiction by asserting that such arguments are not cogent, i.e., they are not arguments which can be put forward for acceptance.

References
Ashcroft, H. (2012, Nov. 20). “L’Aquila Earthquake – shaking the scientific community.” Retrieved from http://www.bangscience.org/2012/11/laquila-earthquake-shaking-the-scientific-community/
Bailin, S. & Battersby, M. (2010). Reason in the balance: An inquiry approach to critical thinking. Whitby, Ont.: McGraw-Hill.
Bailin, S. & Battersby, M. (2009). Inquiry: A dialectical approach to teaching critical thinking. In J. Ritola (Ed.), Argument cultures: Proceedings of OSSA 8, CD-ROM. Windsor, ON: OSSA.
Battersby, M. & Bailin, S. (2011). Guidelines for reaching a reasoned judgment. In J. A. Blair & R. H. Johnson (Eds.). Conductive argument: An overlooked type of defeasible reasoning (pp. 145–157). London: College Publications.
Billi, M. (2013). Sentenza. Tribunale di L’Aquila. Sezione Penale. N.253/2010 R.G.N.R. Retrieved from http://processoaquila.files.wordpress.com/2013/01/sentenza-grandi-rischi-completa-1.pdf
Blair, J. A. & Johnson, R. H. (Eds.). (2011). Conductive argument: An overlooked type of defeasible reasoning. London: College Publications.
Govier, T. (2011). Conductive arguments: overview of the symposium. In J. A. Blair & R. H. Johnson (Eds.). Conductive argument: An overlooked type of defeasible reasoning (pp. 262-276). London: College Publications.
Hitchcock, D. (2002). The practice of argumentative discussion. Argumentation, 6, 3: 287-298.
Hooper, J. (2012, Oct. 22). “Italian scientists convicted for ‘false assurances’ before earthquake.” Retrieved from http://www.theguardian.com/world/2012/oct/22/italian-scientists-jailed-earthquake-aquila
Johnson, R H. (2000). Manifest rationality: A pragmatic theory of argument. Mahwah, NJ: Erlbaum.
Pappas, S. (2012). “Scientists on trial for failing to predict Italian quake.” Retrieved from http://www.nbcnews.com/id/44596501/ns/technology_and_science-science/t/scientists-trial-failing-predict-italian-quake/#.U3J_LF69zw2
US Department of Health and Human Services. (2006). “The Health Consequences of Involuntary Exposure to Tobacco Smoke: A Report of the Surgeon General.” Retrieved from http://www.ncbi.nlm.nih.gov/books/NBK44324/
Watt, S. (2011, Sept. 16).“Scientists in the dock over L’Aquila earthquake.” Retrieved from http://news.bbc.co.uk/2/hi/programmes/newsnight/9593123.stm
Wellman, C. (1971). Challenge and response: Justification in ethics. Carbondale: Southern Illinois University Press.