“However, at the end of the day, no matter what changes are made, scientific credibility may not improve unless the pursuit of truth remains our main goal in our work as scientists. This is a most noble mission that needs to be continuously reasserted.”
Today’s paper: Ioannidis, John. (2012). Why Science is Not Necessarily Self-Correcting.
The ability to self-correct is considered a hallmark of science. However, self-correction does not always happen to scientific evidence by default. The trajectory of scientific credibility can fluctuate over time, both for defined scientific fields and for science at-large. History suggests that major catastrophes in scientific credibility are unfortunately possible and the argument that “it is obvious that progress is made” is weak. Careful evaluation of the current status of credibility of various scientific fields is important in order to understand any credibility deficits and how one could obtain and establish more trustworthy results. Efficient and unbiased replication mechanisms are essential for maintaining high levels of scientific credibility. Depending on the types of results obtained in the discovery and replication phases, there are different paradigms of research: optimal, self-correcting, false nonreplication, and perpetuated fallacy. In the absence of replication efforts, one is left with unconfirmed (genuine) discoveries and unchallenged fallacies. In several fields of investigation, including many areas of psychological science, perpetuated and unchallenged fallacies may comprise the majority of the circulating evidence. I catalogue a number of impediments to self-correction that have been empirically studied in psychological science. Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results as well as some potential disadvantages of each of them. Any deviation from the principle that seeking the truth has priority over any other goals may be seriously damaging to the self-correcting functions of science.
Notes & Quotes
The best thing about this paper is the fictional story written about a faraway planet named F345, in the year 3,045,268. It makes the case better than anything else, and is worth reading in its entirety. I’ll give a few snippets.
Planet F345 in the Andromeda galaxy is inhabited by a highly intelligent humanoid species very similar to Homo sapiens sapiens.
He goes on to describe a situation where science on this planet has become a total joke:
No one is interested in replicating anything in F345. Replication is considered a despicable exercise suitable only for idiots capable only of me-too mimicking, and it is definitely not serious science.
Which is, of course, the opposite of how it should be.
Simple citizens are bombarded from the mass media on a daily basis with announcements about new discoveries, although no serious discovery has been made in F345 for many years now. Critical thinking and questioning is generally discredited in most countries in F345.
Double speak lives on:
As a result, for example, the high-est salaries for scientists and the most sophisticated research infrastructure are to be found in totalitarian countries with lack of freedom of speech or huge social inequalities — one of the most common being gender inequalities against men (e.g., men cannot drive a car and when they appear in public their whole body, including their head, must be covered with a heavy pink cloth). Science is flourishing where free thinking and critical questioning are rigorously restricted…
If all of this doesn’t sound eerily familiar, perhaps you haven’t been paying attention.
The good news is that this kind of critique has not been falling on deaf ears. The bad news is twofold: 1) most parties are not agreed on what the solutions should be (some ideas are much better than others); 2) it will likely take (at least) one generation before the necessary changes in methodology and culture become the new norm. But that is bad news that can be overcome. So it’s also worth staying optimistic.
Among the many changes on the table are:
- better statistical education for scientists in the health and human sciences;
- replacing or updating how p-values are used;
- rejecting “statistical significance” as the litmus test for publication;
- making replication a primary (and rewarded) goal for researchers;
All of these are worth pushing for. Today’s paper was from 2012, and four years has proven to be a long time in this struggle. I think there are clear signs of things moving in the right direction. But, as I said above, it will likely take an entire generation before the problems are no longer the norm.
More important than fixing what’s broken today is instilling a culture that consistently watch-dogs itself to prevent this kind of thing from happening. The point Ioannidis is trying to make is that science is not inherently self-correcting. We have to make self-correction a goal in-itself.
Now go lift something heavy,