 Jens Lasthein/Handout via REUTERSTHE NEWS An astonishing number of scientific studies, especially in medicine and the social sciences, are wrong. Statistical naïveté, poor practice, and outright fraud have meant that scientific journals have filled up with false information. Systematic attempts to replicate the findings of old studies have failed in between a half and two-thirds of cases. Psychology has been the field most shaken by the “replication crisis,” as it’s known. Many foundational texts are now known to be false, or at the least cannot be replicated. For instance, “power posing,” the finding that adopting “powerful” stances makes you feel more confident, became hugely influential — a TED talk on the subject had 70 million views — but has failed to be replicated. Other sciences have been undermined too. In biomedicine, a 2021 project was able to replicate fewer than half of the early-stage cancer biology studies it looked at. Brain imaging studies are usually too weak, statistically speaking, to detect the things they claim to have found. The stakes are as high as they get: If we can’t trust science — to cure diseases, improve our lives, fix the planet — what can we trust? Over the last decade, scientists have been grappling with the problems in their disciplines, and trying to find ways to overcome them. KNOW MORE Academics are judged on how many papers they can publish in high-impact journals — so much so that their career path is sometimes called “publish or perish.” And journals, usually, only publish exciting, novel, positive results. One way of “finding” positive results is to make up your data — outright fraud: More than 8,000 biomedical research papers have been retracted for suspected fraud since 2003. Others plagiarize data and text from older studies. You can even just torture it, and it’ll tell you anything you want to hear. In 2011, three researchers published a paper which found — to the level of “statistical significance” which most journals demand — that listening to When I’m Sixty-Four by The Beatles made people younger. Not metaphorically younger. Literally younger. Which is, of course, impossible. They did this as a stunt, to demonstrate that using perfectly normal statistical practice — in particular, measuring lots of different things and seeing if there were any interesting correlations, a practice called “p-hacking” — you could find almost anything you liked. Taken together, journals’ demand for novelty and the ubiquity of p-hacking made it essentially inevitable that the results of most published scientific papers in many disciplines would be false. TOM’S VIEW Scientists have begun to take some promising steps to resolve this crisis. For one, researchers are increasingly “preregistering” results: writing down in advance what they’re going to look at in their data, so they can’t, afterward, chop it up in a hundred different ways until they find something. Some journals now also promise that they’ll publish studies, not on the basis of their results — that is, whether they’re novel or not, incentivizing scientists to find those novel results — but on the basis of how the study will be done, before the results are collected. A 2021 study found that these kinds of studies, known as “Registered Reports,” were half as likely to find positive results, suggesting that half of the positive results in the other studies were false. But it’s slow progress. Many scientists, especially older ones, resent “data thugs” combing over their old work and showing it to be flawed or fraudulent. Most journals do not require preregistration, let alone promise publication in advance. They often refuse to publish failed replications, meaning that flawed studies go uncorrected. One journal published a paper in 2011 apparently showing that psychic powers were real — and then refused to publish three other scientists’ failed attempt to replicate it. Science is still the best — arguably the only — real route to reliable knowledge of the world around us. We can send spacecraft to other planets, live twice as long as our predecessors, and talk to anyone in the world with handheld devices because of science. But it’s also a human endeavor and flawed in human ways. Scientists want to get promoted and feed their kids the same as the rest of us, and like the rest of us, they’ll cut corners and fudge results to do so. The anti-vaccine movement of the last few decades has shown us what a lack of trust in science can do. It’s vital, therefore, that science gets its house in order. NOTABLE - The incentive structure of science can lead to more outright breaches than relatively subtle p-hacking and HARKing. Fabrication of data and plagiarism are common too: The Economist reported that there is a “worrying amount of fraud” in medical research, and surprisingly little done about it.
| 
