A Field Guide to Critical Thinking

V - A Discordian is Prohibited of Believing what he reads.
- The Five Commandments (the Pentabarf), page 00004, the Principia Discordia


A Field Guide to Critical Thinking
by James Lett

There are many reasons for the popularity of paranormal beliefs in the United States today, including:

* the irresponsibility of the mass media, who exploit the public taste for nonsense,
* the irrationality of the American world-view, which supports such unsupportable claims as life after death and the efficacy of the polygraph, and
* the ineffectiveness of public education, which generally fails to teach students the essential skills of critical thinking.

As a college professor, I am especially concerned with this third problem. Most of the freshman and sophomore students in my classes simply do not know how to draw reasonable conclusions from the evidence. At most, they've been taught in high school what to think; few of them know how to think.

In an attempt to remedy this problem at my college, I've developed an elective course called "Anthropology and the Paranormal." The course examines the complete range of paranormal beliefs in contemporary American culture, from precognition and psychokinesis to channeling and cryptozoology and everything between and beyond, including astrology, UFOs, and creationism. I teach the students very little about anthropological theories and even less about anthropological terminology. Instead, I try to communicate the essence of the anthropological perspective, by teaching them, indirectly, what the scientific method is all about. I do so by teaching them how to evaluate evidence. I give them six simple rules to follow when considering any claim, and then show them how to apply those six rules to the examination of any paranormal claim.

The six rules of evidential reasoning are my own distillation and simplification of the scientific method. To make it easier for students to remember these half-dozen guidelines, I've coined an acronym for them: Ignoring the vowels, the letters in the word "FiLCHeRS" stand for the rules of Falsifiability, Logic, Comprehensiveness, Honesty, Replicability, and Sufficiency. Apply these six rules to the evidence offered for any claim, I tell my students, and no one will ever be able to sneak up on you and steal your belief. You'll be filch-proof.

Falsifiability

It must be possible to conceive of evidence that would prove the claim false. It may sound paradoxical, but in order for any claim to be true, it must be falsifiable. The rule of falsifiability is a guarantee that if the claim is false, the evidence will prove it false; and if the claim is true, the evidence will not disprove it (in which case the claim can be tentatively accepted as true until such time as evidence is brought forth that does disprove it). The rule of falsifiability, in short, says that the evidence must matter, and as such it is the first and most important and most fundamental rule of evidential reasoning.

The rule of falsifiability is essential for this reason: If nothing conceivable could ever disprove the claim, then the evidence that does exist would not matter; it would be pointless to even examine the evidence, because the conclusion is already known -- the claim is invulnerable to any possible evidence. This would not mean, however, that the claim is true; instead it would mean that the claim is meaningless. This is so because it is impossible -- logically impossible -- for any claim to be true no matter what. For every true claim, you can always conceive of evidence that would make the claim untrue -- in other words, again, every true claim is falsifiable.

For example, the true claim that the life span of human beings is less than 200 years is falsifiable; it would be falsified if a single human being were to live to be 200 years old. Similarly, the true claim that water freezes at 32° F is falsifiable; it would be falsified if water were to freeze at, say, 34° F. Each of these claims is firmly established as scientific "fact," and we do not expect either claim ever to be falsified; however, the point is that either could be. Any claim that could not be falsified would be devoid of any propositional content; that is, it would not be making a factual assertion -- it would instead be making an emotive statement, a declaration of the way the claimant feels about the world. Nonfalsifiable claims do communicate information, but what they describe is the claimant's value orientation. They communicate nothing whatsoever of a factual nature, and hence are neither true nor false. Nonfalsifiable statements are propositionally vacuous.

There are two principal ways in which the rule of falsifiability can be violated -- two ways, in other words, of making nonfalsifiable claims. The first variety of nonfalsifiable statements is the undeclared claim: a statement that is so broad or vague that it lacks any propositional content. The undeclared claim is basically unintelligible and consequently meaningless. Consider, for example, the claim that crystal therapists can use pieces of quartz to restore balance and harmony to a person's spiritual energy. What does it mean to have unbalanced spiritual energy? How is the condition recognized and diagnosed? What evidence would prove that someone's unbalanced spiritual energy had been -- or had not been -- balanced by the application of crystal therapy? Most New Age wonders, in fact, consist of similarly undeclared claims that dissolve completely when exposed to the solvent of rationality.

The undeclared claim has the advantage that virtually any evidence that could be adduced could be interpreted as congruent with the claim, and for that reason it is especially popular among paranormalists who claim precognitive powers. Jeane Dixon, for example, predicted that 1987 would be a year "filled with changes" for Caroline Kennedy. Dixon also predicted that Jack Kemp would "face major disagreements with the rest of his party" in 1987 and that "world-wide drug terror" would be "unleashed by narcotics czars" in the same year. She further revealed that Dan Rather "may [or may not] be hospitalized" in 1988, and that Whitney Houston's "greatest problem" in 1986 would be "balancing her personal life against her career." The undeclared claim boils down to a statement that can be translated as "Whatever will be, will be."

The second variety of nonfalsifiable statements, which is even more popular among paranormalists, involves the use of the multiple out, that is, an inexhaustible series of excuses intended to explain away the evidence that would seem to falsify the claim. Creationists, for example, claim that the universe is no more than 10,000 years old. They do so despite the fact that we can observe stars that are billions of light-years from the earth, which means that the light must have left those stars billions of years ago, and which proves that the universe must be billions of years old. How then do the creationists respond to this falsification of their claim? By suggesting that God must have created the light already on the way from those distant star at the moment of creation 10,000 years ago. No conceivable piece of evidence, of course, could disprove that claim.

Additional examples of multiple outs abound in the realm of the paranormal. UFO proponents, faced with a lack of reliable physical or photographic evidence to buttress the claims, point to a secret "government conspiracy" that is allegedly preventing the release of evidence that would support their case. Psychic healers say they can heal you if you have enough faith in their psychic powers. Psychokinetics say they can bend spoons with their minds if they are not exposed to negative vibrations from skeptic observers. Tarot readers can predict your fate if you're sincere in your desire for knowledge. The multiple out means, in effect, "Heads I win, tails you lose."

Logic

Any argument offered as evidence in support of any claim must be sound. An argument is said to be "valid" if its conclusion follows unavoidably from its premises; it is "sound" if it is valid and if all the premises are true. The rule of logic thus governs the validity of inference. Although philosophers have codified and named the various forms of valid arguments, it is not necessary to master a course in form logic in order to apply the rules of inference consistently and correctly. An invalid argument can be recognize by the simple method of counterexample: If you can conceive of a single imaginable instance whereby the conclusion would not necessarily follow from the premises even if the premises were true, then the argument is invalid. Consider the following syllogism for example: All dogs have fleas; Xavier has fleas; therefore Xavier is a dog. That argument is invalid because a single flea-ridden feline named Xavier would provide an effective counterexample. If an argument is invalid, then it is, by definition, unsound. Not all valid arguments are sound, however. Consider this example: All dogs have fleas; Xavier is a dog; therefore Xavier has fleas. That argument is unsound, even though it is valid, because the first premise is false: All dogs do not have fleas.

To determine whether a valid argument is sound is frequently problematic; knowing whether a given premise is true or false often demands additional knowledge about the claim that may require empirical investigation. If the argument passes these two tests, however -- if it is both valid and sound -- then the conclusion can be embraced with certainty.

The rule of logic is frequently violated by pseudoscientists. Erich von Däniken, who singlehandedly popularized the ancient-astronaut mythology in the 1970s, wrote many books in which he offered invalid and unsound arguments with benumbing regularity (see Omohundro 1976). In Chariots of the Gods? he was not above making arguments that were both logically invalid and factually inaccurate -- in other words, arguments that were doubly unsound. For example, von Däniken argues that the map of the world made by the sixteenth-century Turkish admiral Piri Re'is is so "astoundingly accurate" that it could only have been made from satellite photographs. Not only is the argument invalid (any number of imaginable techniques other than satellite photography could result in an "astoundingly accurate" map), but the premise is simply wrong -- the Piri Re'is map, in fact, contains many gross inaccuracies (see Story 1981).

Comprehensiveness

The evidence offered in support of any claim must be exhaustive -- that is all of the available evidence must be considered.

For obvious reasons, it is never reasonable to consider only the evidence that supports a theory and to discard the evidence that contradicts it. This rule is straightforward and self-apparent, and it requires little explication or justification. Nevertheless, it is a rule that is frequently broken by proponents of paranormal claims and by those who adhere to paranormal beliefs.

For example, the proponents of biorhythm theory are fond of pointing to airplane crashes that occurred on days when the pilot, copilot, anchor navigator were experiencing critically low points in their intellectual, emotional, and/or physical cycles. The evidence considered by the biorhythm apologists, however, does not include the even larger number of airplane crashes that occurred when the crews were experiencing high or neutral points in their biorhythm cycles (Hines 1988:160). Similarly, when people believe that Jeane Dixon has precognitive ability because she predicted the 1988 election of George Bush (which she did, two months before the election, when every social scientist, media maven, and private citizen in the country was making the same prognostication), they typically ignore the thousands of forecasts that Dixon has made that have failed to come true (such as her predictions that John F. Kennedy would not win the presidency in 1960, that World War III would begin in 1958, and that Fidel Castro would die in 1969). If you are willing to be selective in the evidence you consider, you could reasonably conclude that the earth is flat.

Honesty

The evidence offered in support of any claim must be evaluated without self-deception.

The rule of honesty is a corollary to the rule of comprehensiveness. When you have examined all of the evidence, it is essential that you be honest with yourself about the results of that examination. If the weight of the evidence contradicts the claim, then you are required to abandon belief in that claim. The obverse, of course, would hold as well.

The rule of honesty, like the rule of comprehensiveness, is frequently violated by both proponents and adherents of paranormal beliefs. Parapsychologists violate this rule when they conclude, after numerous subsequent experiments have failed to replicate initially positive psi results, that psi must be an elusive phenomenon. (Applying Occam's Razor, the more honest conclusion would be that the original positive result must have been a coincidence.) Believers in the paranormal violate this rule when they conclude, after observing a "psychic" surreptitiously bend a spoon with his hands, that he only cheats sometimes.

In practice, the rule of honesty usually boils down to an injunction against breaking the rule of falsifiability by taking a multiple out. There is more to it than that, however: The rule of honesty means that you must accept the obligation to come to a rational conclusion once you have examined all the evidence. If the overwhelming weight of all the evidence falsifies your belief, then you must conclude that the belief is false, and you must face the implications of that conclusion forthrightly. In the face of overwhelmingly negative evidence, neutrality and agnosticism are no better than credulity and faith. Denial, avoidance, rationalization, and all the other familiar mechanisms of self-deception would constitute violations of the rule of honesty.

In my view, this rule alone would all but invalidate the entire discipline of parapsychology. After more than a century of systematic, scholarly research, the psi hypothesis remains wholly unsubstantiated and unsupportable; parapsychologists have failed, as Ray Hyman (1985:7) observes, to produce "any consistent evidence for paranormality that can withstand acceptable scientific scrutiny." From all indications, the number of parapsychologists who observe the rule of honesty pales in comparison with the number who delude themselves. Veteran psychic investigator Eric Dingwall (1985:162) summed up his extensive experience in parapsychological research with this observation: "After sixty years' experience and personal acquaintance with most of the leading parapsychologists of that period I do not think I could name a half dozen whom I could call objective students who honestly wished to discover the truth."

Replicability

If the evidence for any claim is based upon an experimental result, or if the evidence offered in support of any claim could logically be explained as coincidental, then it is necessary for the evidence to be repeated in subsequent experiments or trials.

The rule of replicability provides a safeguard against the possibility of error, fraud, or coincidence. A single experimental result is never adequate in and of itself, whether the experiment concerns the production of nuclear fusion or the existence of telepathic ability. Any experiment, no matter how carefully designed and executed, is always subject to the possibility of implicit bias or undetected error. The rule of replicability, which requires independent observers to follow the same procedures and to achieve the same results, is an effective way of correcting bias or error, even if the bias or error remains permanently unrecognized. If the experimental results are the product of deliberate fraud, the rule of replicability will ensure that the experiment will eventually be performed by honest researchers.

If the phenomenon in question could conceivably be the product of coincidence, then the phenomenon must be replicated before the hypothesis of coincidence can be rejected. If coincidence is in fact the explanation for the phenomenon, then the phenomenon will not be duplicated in subsequent trials, and the hypothesis of coincidence will be confirmed; but if coincidence is not the explanation, then the phenomenon may be duplicated, and an explanation other than coincidence will have to be sought. If I correctly predict the next roll of the dice, you should demand that I duplicate the feat before granting that my prediction was anything but a coincidence.

The rule of replicability is regularly violated by parapsychologists, who are especially fond of misinterpreting coincidences. The famous "psychic sleuth" Gerard Croiset, for example, allegedly solved numerous baffling crimes and located hundreds of missing persons in a career that spanned five decades, from the 1940s until his death in 1980. The truth is that the overwhelming majority of Croiset's predictions were either vague and nonfalsifiable or simply wrong. Given the fact that Croiset made thousands of predictions during his lifetime, it is hardly surprising that he enjoyed one or two chance "hits." The late Dutch parapsychologist Wilhelm Tenhaeff, however, seized upon those "very few prize cases" to argue that Croiset possessed demonstrated psi powers (Hoebens 1986a:130). That was a clear violation of the rule of replicability, and could not have been taken as evidence of Croiset's psi abilities even if the "few prize cases" had been true. (In fact, however, much of Tenhaeff's data was fraudulent -- see Hoebens 1986b. )

Sufficiency

The evidence offered in support of any claim must be adequate to establish the truth of that claim, with these stipulations:

* the burden of proof for any claim rests on the claimant,
* extraordinary claims demand extraordinary evidence, and
* evidence based upon authority and/or testimony is always inadequate for any paranormal claim

The burden of proof always rests with the claimant for the simple reason that the absence of disconfirming evidence is not the same as the presence of confirming evidence. This rule is frequently violated by proponents of paranormal claims, who argue that, because their claims have not been disproved, they have therefore been proved. (UFO buffs, for example, argue that because skeptics have not explained every UFO sighting, some UFO sightings must be extraterrestrial spacecraft.) Consider the implications of that kind of reasoning: If I claim that Adolf Hitler is alive and well and living in Argentina, how could you disprove my claim? Since the claim is logically possible, the best you could do (in the absence of unambiguous forensic evidence) is to show that the claim is highly improbable -- but that would not disprove it. The fact that you cannot prove that Hitler is not living in Argentina, however, does not mean that I have proved that he is. It only means that I have proved that he could be -- but that would mean very little; logical possibility is not the same as established reality. If the absence of disconfirming evidence were sufficient proof of a claim, then we could "prove" anything that we could imagine. Belief must be based not simply on the absence of disconfirming evidence but on the presence of confirming evidence. It is the claimant's obligation to furnish that confirming evidence.

Extraordinary claims demand extraordinary evidence for the obvious reason of balance. If I claim that it rained for ten minutes on my way to work last Tuesday, you would be justified in accepting that claim as true on the basis of my report. But if I claim that I was abducted by extraterrestrial aliens who whisked me to the far side of the moon and performed bizarre medical experiments on me, you would be justified in demanding more substantial evidence. The ordinary evidence of my testimony, while sufficient for ordinary claims, is not sufficient for extraordinary ones.

In fact, testimony is always inadequate for any paranormal claim, whether it is offered by an authority or a layperson, for the simple reason that a human being can lie or make a mistake. No amount of expertise in any field is a guarantee against human fallibility, and expertise does not preclude the motivation to lie; therefore a person's credentials, knowledge and experience cannot, in themselves be taken as sufficient evidence to establish the truth of a claim. Moreover, a person's sincerity lends nothing to the credibility of his or her testimony. Even if people are telling what they sincerely believe to be the truth, it is always possible that they could be mistaken. Perception is a selective act, dependent upon belief context, expectation, emotional and biochemical states, and a host of other variables. Memory is notoriously problematic, prone to a range of distortions, deletions, substitutions and amplifications. Therefore the testimony that people offer of what they remember seeing or hearing should always be regarded as only provisionally and approximately accurate; when people are speaking about the paranormal, their testimony should never be regarded as reliable evidence in and of itself. The possibility and even the likelihood of error are far too extensive (see Connor 1986).

Conclusion

The first three rules of FiLCHeRS -- falsifiability, logic, and comprehensiveness -- are all logically necessary rules of evidential reasoning. If we are to have confidence in the veracity of any claim whether normal or paranormal, the claim must be prepositionally meaningful, and the evidence offered in support of the claim must be rational and exhaustive.

The last three rules of FiLCHeRS -- honesty, replicability, and sufficiency -- are all pragmatically necessary rules of evidential reasoning. Because human beings are often motivated to rationalize and to lie to themselves, because they are sometimes motivated to lie to others, because they can make mistakes, and because perception and memory are problematic, we must demand that the evidence for any factual claim be evaluated without self-deception, that it be carefully screened for error, fraud, and appropriateness, and that it be substantial and unequivocal.

What I tell my students, then, is that you can and should use FiLCHeRS to evaluate the evidence offered for any claim. If the claim fails any one of these six tests, then it should be rejected; but if it passes all six tests, then you are justified in placing considerable confidence in it.

Passing all six tests, of course, does not guarantee that the claim is true (just because you have examined all the evidence available today is no guarantee that there will not be new and disconfirming evidence available tomorrow), but it does guarantee that you have good reasons for believing the claim. It guarantees that you have sold your belief for a fair price, and that it has not been filched from you.

Being a responsible adult means accepting the fact that almost all knowledge is tentative, and accepting it cheerfully. You may be required to change your belief tomorrow, if the evidence warrants, and you should be willing and able to do so. That, in essence, is what skepticism means: to believe if and only if the evidence warrants.