Reproducibility: standardized vs. heterogenized replicates

Here's a paper that caught Hendrik Poorter's attention awhile ago.

In ecology, we almost never run the same experiment twice. We reason that resources are just too thin to try to reproduce someone's results.

Even if we did try the same experiment twice, there are enough differences among ecosystems to explain any differences.

But, if we did try the same experiment twice, how do we set up our experiment to maximize reproducibility?

Reproducibility is simply the ability to generate the same results twice. If we want to affirm generalization in our results, experiments should be reproducible.

The Richter et al. paper from 2009 addressed this question for the biomedical world. There they noted that experiments with animals are costly, so there should be an impetus to make results be as reproducible as possible.

The natural approach to this is to make experiments as controlled and uniform as possible.

This reduces variance among subjects of a given experiment and maximizes the likelihood of a significant result, but actually works to reduce reproducibility.

In short, if you want reproducibility, experiments need to incorporate all the variation that one is likely to encounter when trying to reproduce experiments.

Differences in users.
Differences in noise.
Differences in cage types.

Instead of controlling for all of these, they argue that the experiments should allow these to vary. In doing so, any positive result is more likely to be generalizable.

Instead of using "standardized" replicates, replicates should be "heterogenized".

What is interesting is that the same holds for many ecological experiments. If we want to make sure we do not have a false positive, we should incorporate variation into our experiments.

It's risky for the investigator, because the approach reduces the likelihood of a positive result.

But for the discipline, it makes it more likely that any one result is not only reproducible, but also generalizable.