What-About-This syndrome: proof positive?

Outrunning the parsimony bear.

There's the old joke about two campers being chased by a bear. The discussion between the two campers as they flee the bear typically involves one camper putting on running shoes and centers around whether it is necessary to outrun the bear or outrun the other camper. The implication being that the bear will eat whoever is slower.

I think about this joke when I read scientific reviews of manuscripts. These are critical junctures for scientific progress and lay bare the scientific process. Review is also the most unregulated part of the scientific process. Assuming no methodological flaws, judging scientific merit of papers is not unlike campers being chased by a bear. The question of whether the one camper needs to outrun the bear or the other camper is similar to whether a hypothesis is "proven" (outruns the bear) or whether it is the most parsimonious (outruns the other camper).

One of the common patterns of reviewers is "What-About-This?" syndrome. I first noticed this in EcoLunch at Berkeley when one member seemed to almost invariably to ask the presenter about an extreme point on a scatter plot (What about that point? What happened there?). There is merit to examining individual cases, but outliers never disprove generalities.

Often, a reviewer will raise an issue or an alternative hypothesis not discussed by the authors. It's impossible for authors to raise every issue or alternative hypothesis, so these comments are always possible. Generally, the reviewer will not describe how this hypothesis is more parsimonious or how it invalidates the support for the hypothesis. For a paper to be rejected by the simple raising of a "what-about-this" point implies that all competing hypotheses have to be disproved.

Theoretically, when we test hypotheses, we seek out the most parsimonious explanation. This is codified in some statistical tests, e.g. AIC model testing. This is codified in the way we reconstruct phylogenetic trees. Yet, often when we evaluate whether a paper should enter the scientific literature, it is as if a different set of rules applies.

The question that has never been answered with regard to reviewing is whose role is it to evaluate parsimony? If a reviewer raises a what-about-this that could serve as an alternative hypothesis, is it their responsibility to show that is also likely more parsimonious? Or is it then the author's role? If the latter, then the editor cannot allow a paper to be rejected on scientific grounds based on what-about-this's.

Parsimony is a slippery topic, but so is proof. We need to promote discussions of parsimony not only for the mechanics of science, but also for its application. When the debate becomes whether we have proved that the Earth is warming, that debate is already lost. A common tactic is to raise what-about-this's and use that as evidence denying proof.

Application of science needs to engender parsimony. But so does the scientific process itself.

The footrace in science is never about crossing the finish line, only staying ahead of the other runners.