The process of science is one we do not talk about much. There are reams of studies on statistical tests for a given data set, and meta-analyses have moved science forward for bringing together different data sets to test an idea.
But how does science decide the "truth" when there are different assumptions between different studies? What process gets used when words are used in different ways? No statistical test or meta-analysis can bridge that gap.
Here's an example...
In August of 2014, Michaletz et al. published a paper that analyzed data on plant production for over a thousand forests across the world. It has long been understood that production is greatest in warm, wet forests (think tropical rain forests) and least in cold, dry forests (think bristlecone pine).When we warm or irrigate forests, they grow more, too. Seems like the role of climate is pretty well settled.
In their review, the authors found that, indeed, production correlated with temperature and precipitation, but according to them, this was too simple. When viewed through metabolic scaling theory, climate had only an indirect effect on production. The authors asserted that "age and biomass explained most of the variation in production whereas temperature and precipitation explained almost none". In short, warm, wet forests are more productive only because they tend to be older and larger there, not because warm or wet conditions promote growth. By this idea, if you compare two forests of equal size and age, but one forest was in a cold, dry environment, and the other was in a warm, wet environment, there would be no difference in their production.
The authors have published many excellent papers on metabolic scaling, really developing a line of thought to begin to unify some fractured thought on how plants work. If this result held, it would be a coup de grace in many ways.
So how did the authors rule out that climate directly affected production?
The authors calculated a rate of monthly production by dividing production by the length of the growing season. This removed the influence of differences in the length of the growing season to compare forests across the world more equally, essentially asking if forests in warm, wet places grow more each month than ones in cold, dry places. When they did this, they found that "In contrast to results for NPP, average growing season temperature,...mean annual precipitation, and mean growing season precipitation explained little to no variation in global [monthly production]."
And with that result, the authors move on to test other factors, such as stand age and biomass, independent of climate, finding that "A large proportion of variation in NPP...was explained by just two variables: stand biomass and plant age."
The Michaletz paper was published in Nature, which is often publishes some of the most important results in our discipline only after intense scrutiny. It seemed like that question was settled. Climate only affects how big forests get and how old they are, it doesn't make a given forest grow any faster per se.
Well, I guess it can be said that one person's assumption is another person's legerdemain.
This past January a new paper was published in Global Change Biology. Chu et al. reanalyze the Michaletz data and start with the title "Does climate directly influence NPP globally?" The authors assert that the Michaletz study had "flaws that affected that study’s conclusions". They also "present novel analyses to disentangle the effects of stand variables and climate in determining NPP."
In short, the authors state that ruling out climate's direct effect by calculating monthly production was erroneous. Growing season length and mean climate are highly correlated. In their view, it was incorrect to rule out the direct effect of climate by dividing production by growing season length and then examining the resultant metric against climate variables.**
**This debate, in part, is the Knops-Vitousek debate all over again...
Instead, using different analytic techniques, Chu et al. simultaneously test the roles of growing season length and other climate variables on production.
Their conclusion? Climate does directly affects production.
At this point, I'm not writing about this to weigh in on which side is right or more right or right under specific conditions.
I only pose this question.
How does our discipline resolve the tension here?
Were the assumptions by Michaletz right? Are the two camps' differences semantic? Which conclusion should be accepted? Does climate directly affect production or only indirectly?
At this point, if it was convenient for a scientist's argument for climate not to affect production, they just cite the Michaletz paper. If the contrary held, just cite the Chu paper.
In the legal world, when different circuit courts come to different conclusions, this can lead to "forum shopping" where a plaintiff can simply go to the circuit that is most favorable to their case. That shouldn't be if the goal is to have one set of laws to govern a nation.
Like the legal world, it seems like being able to cite either one of two opposing ideas is not sustainable for science either.
It is interesting to note that in the US federal court system, two contrary ideas existing at the same time would be the equivalent of a "circuit split" where two circuit courts come to two different conclusions about how to interpret the law. This tension would often be resolved at the next higher court, the Supreme Court. And the decision of the Supreme Court would resolve the differences of opinion.
All I note here is that science doesn't have that. We have no formalized process for resolving a split. Split conclusions can theoretically last indefinitely. And scientists can cite whichever side they believe in more or find most convenient.
I think that is fascinating.