By Russell K. King, HHRA Executive Director
Less than a week ago, the academic publisher Sage Journals retracted studies that questioned the long-established safety record of mifepristone. In December. Nature ran a piece noting that, in 2023, more than 10,000 scientific papers–a record number–were retracted. Not understanding the data, the anti-science voices decry the retractions as proof of corruption in the research community. Misunderstanding scientific publishing is an old and common problem. Early in my career, I was editor of a peer-reviewed medical journal, and part of my job was to translate the scientific language into messages more easily understood by nonscientific readers. I offer here a guide to reading scientific papers when you’re not a scientist. This method is not the only method, and I didn’t create it, but I’ve found it useful.
Before I do, however, I hasten to say that the wave of retractions last year does not indicate a wave of fraud in science. The number of journals in publication rose from 1 million in 1997 to 3 million in 2020, yet the average number of retractions per journal has remained largely flat during that time. Half of the retractions are for reasons other than fabrication, falsification, or plagiarism. The data seem to say the scientific community has stepped up. Scientists are pressuring journals, and, in turn, journals are improving their policing of papers after publication.
Reading scientific papers
Step by step:
1. read the abstract to get the general idea of what the paper is about;
2. read the figures and legends to understand the data (then look to see whether they align with the conclusions in the abstract);
3. read the discussion, where the authors summarize and interpret the data (then see whether it aligns with the data in the figures and the overview in the abstract); and
4. if it’s not your field of expertise (true for most of us, even if it’s hard to admit), read the introduction to get a feel for what the relevant literature says;
5. if you’re evaluating how they got from the data to the conclusions, read the discussion (are they using standard methods, missing controls, using a representative sample and a control group, etc.?); and
6. read through a few references to see whether they say what the authors claim they do (padding the references with papers that do not fit is often a way to build false credibility).
Red flags
As you’re reading, keep your eyes open for signs that should cause you to pause and question the paper’s validity. Red flags don’t necessarily mean the paper is untrustworthy, just that we should not draw conclusions without digging deeper. Some such red flags are:
1. the author has no expertise in the subject of the paper (is their degree in a relevant field, have they worked in the field, have they previously published in the field in reputable journals?)–crossovers are not uncommon, but these will typically have a coauthor who has credible expertise;
2. the references are old, meaning fewer than six citations from the past five years;
3. the results asserted are not closely tied to the data or are not placed in context with other studies;
4. the conclusions contradict the literature or general scientific consensus–advances happen, but this should prompt us to withhold judgment until we get more information;
5. funders are not disclosed;
6. conflicts of interest are not declared; and
7. the results have not been peer reviewed.
Know yourself
Because science and anti-science have become such powerful forces in cultural and political differences, it’s vital that we check ourselves as we read and evaluate scientific papers. No matter how well trained we are in critical thinking, no matter how separate we think we are from the cultural and political echo chambers around us, we are still human and we are still given to myriad thinking errors. To deal with the overwhelming amount of information our brains take in, our brains seek shortcuts to lessen the burden. Sometimes these shortcuts are helpful; too often they are not. At minimum, we process information through our personal confirmation bias and a complex, overlapping, ever-changing matrix of internal filters made of everything from our DNA to what we had for lunch.
We must ask ourselves–more often than is comfortable–whether our understanding of what we’re reading is being distorted by our own emotions, preferences, prejudices, assumptions, and hopes. This requires us to be honest with ourselves about our emotions, preferences, prejudices, assumptions, and hopes.
Read!
Science is always emerging, never static. By the moment, it grows ever broader, deeper, more beautiful, more fascinating, and more important to our lives. We depend on science to bring us new information and understanding, to correct the errors of our past and–yes–to retract papers that are erroneous. The 10,000 retractions of 2023 should enhance, not undermine, our appreciation for the men and women of science who share their work with us. What’s happening in science is exciting, and I promise you that reading about it is more than worth the effort.