How to evaluate scientific studies cited in news
Most science news massively oversimplifies the underlying study. Here's a five-step process for getting closer to what the study actually says.
“A new study finds…” is one of the most-used phrases in news writing and one of the most-misused. Most news coverage of scientific studies oversimplifies, exaggerates, or misframes the underlying research. Here’s the working journalist’s process for evaluating any study cited in news.
1. Find the actual study
News articles should link to the paper. If they don’t, search the lead researcher’s name + the study topic on Google Scholar or PubMed. Read at minimum the abstract and the discussion / limitations section.
If the news article doesn’t link to the paper AND you can’t find it on Scholar or PubMed within a minute, that’s a strong warning sign — the “study” may not exist as the article describes it.
2. Check who funded it
Funding bias is a real and well-documented effect. A study of a drug funded by the drug’s manufacturer is more likely to find favorable results than the same study funded by an independent grant. Funding sources are required to be disclosed in any reputable journal — find the disclosure and weigh it accordingly.
3. Check the study type
Different study types support different strength of conclusions: - Systematic review / meta-analysis: highest evidence weight, combines many studies - Randomized controlled trial (RCT): strongest single-study evidence for causation - Cohort study: tracks people over time; shows correlation, not necessarily causation - Case-control study: looks backward; weaker for causation - Cross-sectional study: snapshot in time; correlation only - Case report: a single example; lowest evidence weight
A news headline reading “X causes Y” based on a cross-sectional study is overclaiming. Causation requires more than correlation.
4. Check the sample size and population
Small samples (n<100 for behavioral studies, n<30 for clinical) produce wide confidence intervals and are easily overturned by replication. Animal studies don’t reliably extrapolate to humans. Studies on one demographic don’t reliably extrapolate to others.
The phrase “mice that received [X] showed [Y]” is not the same as “[X] causes [Y] in humans”. News writing routinely collapses this distinction.
5. Check the limitations section
Every reputable study has a limitations section in which the authors describe what their study *can’t* tell you. Read it. It’s usually where the most honest assessment of the study’s strength lives.
When to be especially skeptical
- Headlines that read “study proves X” (studies rarely prove anything definitively) - Studies that haven’t been peer-reviewed yet (preprints are useful but not final) - Effect sizes that aren’t mentioned in the article (small effects + statistical significance = often newsworthy but not life-changing) - Single-study conclusions that contradict the broader literature - Studies that perfectly confirm a politically-popular position
A practical example
A 2024 study reported that “coffee drinkers live longer”. The actual study: cohort design, n=300k, association between 2-4 cups of coffee per day and ~10% reduction in all-cause mortality, controlled for some confounders but not all. The honest summary: “There’s an association between moderate coffee consumption and reduced mortality in observational data, with the usual caveats about residual confounding.” The news version: “COFFEE EXTENDS YOUR LIFE.” Both descriptions cite the same paper. They’re very different claims.
Prism Source Receipts will surface when an article’s framing of a study materially exceeds what the underlying paper supports.
Related: Primary source · Fact-checking · False balance.