Don’t believe the hype. Public Enemy wasn’t talking directly about health science, but they easily could have been. The claimed effects in the majority of studies in the health (and fitness) science community are NOT as significant as they want you to believe — and they are not being replicated.
John Ioannidis has a new article in PLoS medicine on the continuing problem in the health (and many other) sciences where research-replication is nearly non-existent because of massive opposing pressures:
“The current system values publications, grants, academic titles, and previously accumulated power. Researchers at higher ranks have more papers and more grants. However, scholars at the very top of the ladder (e.g., university presidents) have modest, mediocre, or weak publication and citation records . This might be because their lobbying dexterity compensates for their lack of such credentials, and their success comes at the expense of other worthier candidates who would bring more intellectual rigor and value to senior decision making; equally, it could be because they excel at the bureaucratic work necessary to keep the mind-boggling academic machine going, and their skills enable more scientifically gifted colleagues to concentrate on research. The current system does not reward replication—it often even penalizes people who want to rigorously replicate previous work, and it pushes investigators to claim that their work is highly novel and significant . Sharing (data, protocols, analysis codes, etc.) is not incentivized or requested, with some notable exceptions –. With lack of supportive resources and with competition (“competitors will steal my data, my ideas, and eventually my funding”), sharing becomes even disincentivized. Other aspects of scientific citizenship, such as high-quality peer review, are not valued. Peer review can be a beneficial process, acting as a safety net and a mechanism for augmenting quality. It can also be superficial, lead to only modest improvements of the reviewed work, and allow for the acceptance of blatantly wrong papers ,. That it is so little valued and rewarded is not calculated to encourage its benefits and minimize its harms. ”
He urges these fields to take a page out of the book of the Natural Sciences, especially Physics and Genetics, which have made great strides in making the scientific process more open, collaborative, and ensuring that findings can be replicated and tested.
Here are the talking points from the article (quoted):
- Currently, many published research findings are false or exaggerated, and an estimated 85% of research resources are wasted.
To make more published research true, practices that have improved credibility and efficiency in specific fields may be transplanted to others which would benefit from them—possibilities include the adoption of large-scale collaborative research; replication culture; registration; sharing; reproducibility practices; better statistical methods; standardization of definitions and analyses; more appropriate (usually more stringent) statistical thresholds; and improvement in study design standards, peer review, reporting and dissemination of research, and training of the scientific workforce.
Selection of interventions to improve research practices requires rigorous examination and experimental testing whenever feasible.
Optimal interventions need to understand and harness the motives of various stakeholders who operate in scientific research and who differ on the extent to which they are interested in promoting publishable, fundable, translatable, or profitable results.
Modifications need to be made in the reward system for science, affecting the exchange rates for currencies (e.g., publications and grants) and purchased academic goods (e.g., promotion and other academic or administrative power) and introducing currencies that are better aligned with translatable and reproducible research.
You can read the full PDF here.
Now go lift something heavy,