How do scientists know what they know? In particular, how do scientists confirm or disconfirm their hypotheses on the basis of evidence? If we take a look at most scientific papers today, the answer seems clear-cut: in most modern scientific research, scientists make a hypothesis, gather evidence, compute a certain number known as the p-value, and, on the basis of this number, either confirm or disconfirm their hypothesis. Certainly, this account of scientific knowledge, known as the classical statistical theory of evidence, makes up an overwhelming proportion of modern scientific research—and if we wish merely to understand how most scientists do their work, then this description would certainly suffice.
However, we now know that this description cannot be complete in regards to the question of how scientists know what they know. In recent years, a group of scientists put modern science to the test by reexamining a set of psychological studies with notable results to see if they would replicate. In doing so, they invited working scientists to bet money on whether or not these studies would replicate. Surprisingly, these scientists tended to bet against the veracity of these studies; that is, they were more pessimistic about the replicability of these studies than the papers’ conclusions might suggest. When the results came in, however, it turned out that the scientists were right—most of the studies, about seventy percent, did not replicate.
In effect, it seems as if there is a disconnect between what scientists say is true in their research and what they actually believe to be true; that is to say, what we see reflected in the scientific literature is an incomplete representation of scientific knowledge. In this paper, I argue, by introducing and defending the problem of optional stopping and the base rate fallacy, that the classical statistical theory of evidence does not adequately address the question of how scientists know what they know. In particular, the replication crisis that we see in science today reflects the shortcomings of the classical theory of evidence, and the conclusions that we draw from it do not adequately reflect scientific knowledge.