Always on the lookout for new blogs, I was happy to see a former leader at the National Science Foundation, Jim Olson, start one (↬Tyler Cowen). Based on the formal and didactic style I would say I was not the target audience for it, but it is better then nothing. It may also provide a convenient catalyst for my own thoughts.
For example: Olson’s most recent post is about the replication crisis. He points the finger on verification not being sexy enough for grant funders and academic journals, which is true. But if anything, having more people verify more and more claims in the ever-growing steaming pile of academese would make things seemingly worse, at least in the short term. This is the same kind of thinking that wanted to end medical reversals. You don’t want to end them, you want to make them unnecessary in the first place!
Now, fear of your claim being verified may frighten some researches from shooting from their hip, but unless paired with some sort of immediate punishment it would hardly make for a good stick. And what is preventing the person who made the original claim from demanding verification of the verifiers, and so on, and so forth, ad infinitum?
Olson also recommends more detailed methods, so that replication would be possible in the first place. This has already been implemented as anyone who had to fill out Cell’s never-ending STAR Methods can attest. Nature and Science have similar requirements, and some of them don’t even have a word count limit for the Methods section. Granted, many other journals aren’t as rigorous, but that should help you figure out which journals to follow.
So, instead of asking why we don’t have more people verifying claims, I would ask why we needed verification in the first place. Olson touches upon the core issue, mentioning “the time horizon problem”:
NSF grants run 3-5 years. Tenure clocks run 6-7 years. But scientific truth emerges over decades. We’re optimizing for the wrong timescale.
During my time at NSF, I saw brilliant researchers make pragmatic choices: publish something surprising now (even if it might not hold up) rather than spend two years carefully verifying it. That’s not a moral failing—it’s responding rationally to the incentives we created.
Of course it is about incentives. No amount of verifying will change that. People are chasing after tenure and accolades, not truth, and many a tenured professors shrugged their shoulders at the mansions of straw they had built over the decades. At best, they provided an easy target for a successor in the field to refute, unless of course there is a whole cabal of like-minded researchers protecting the dubious claims. But the default position is that these mansions of straw stay there, moulding and festering, side-tracking post-docs and spamming PubMed searches.
I have no clue what the solution may be. Maybe there is none and this is the equilibrium — let reality provide the final vote. But the status quo feels far from optimal.