2026 · BMJ
Machine learning based screening of potential paper mill publications in cancer research: methodological and cross sectional study
Verdict
UNSCIENTIFIC
Neither the paradigm nor the study inside it passes. The verdict is unscientific.
1·Study methodology · inside the paradigm
UNSCIENTIFIC
Does not follow the scientific method within Institutional Consensus.
2·Paradigm · Institutional Consensus
UNSCIENTIFIC
Fails the tests. The worse layer decides.
How the verdict is decided
The verdict grades both layers: the paradigm this study assumes, and how the study was carried out inside it. Both must be scientific for the whole thing to be scientific. A clean method inside an unscientific paradigm is unscientific. An unscientific method inside a better paradigm is unscientific. A split on either layer is mixed. The worse layer decides.
1·Study methodology
Did this study test its claim with methods that are independent, falsifiable, and non-circular?
UNSCIENTIFIC
This study claims a text-reading program can spot paper mill fakes from titles and abstracts, and that about 10% of 2.6 million cancer papers, including 36% of Chinese ones, are flagged. All three tests fail.
Independently Verifiable
Fail
The program produced a number for each abstract, and the authors called every number above their cutoff a paper mill paper, though nobody saw who wrote any of them.
Falsifiable
Fail
Training and checking used the same kind of controls, so a program that only learned 'Chinese, microRNA, not a top journal' would still pass every check.
Non-Circular
Fail
They chose control countries because Retraction Watch listed no paper mill retractions there, trained on the retractions, then reported the retraction-heavy countries as flagged and the control countries as clean.
Why
They downloaded 38 million PubMed records and kept the ones with cancer words. Retraction Watch had tagged 2,202 of these as paper mill papers, so those became the bad pile. The good pile came from top-decile journals and from Sweden, Finland, Norway and Taiwan, places with few or no such retractions. A text program learned to separate the piles, then was checked on 3,094 papers from online sleuth lists against similar controls. Both the training set and the checks were half bad, half good, while the real corpus is not, so nothing says how many flags are right. They ran it on 2.6 million abstracts and called 9.87% mills. Nobody read a full text, checked an image, or found who wrote a flagged paper. They saw scores.
2·Paradigm · Institutional Consensus
Does the framework this study assumes pass the three tests?
UNSCIENTIFIC
Independently Verifiable
Fail
Papers and retraction notices can be seen, but 'genuine' and 'paper mill' are verdicts from editors and sleuths, and top-journal status is only assumed to mean quality.
Falsifiable
Fail
A faked paper in a famous journal gets called a miss in the system, and a flagged paper that turns out fine gets called merely 'similar', so neither result ends the consensus.
Non-Circular
Fail
Journal prestige and retraction tags serve as the measure of quality, and the program is trained to reproduce those same labels and then report its output as a finding.
Why
The framework says a paper is real or fake depending on whether journals, retraction databases and sleuth communities have labeled it so. What anyone can observe is that papers exist, that some carry retraction notices, and that some abstracts read alike. The verdicts of genuine and fraudulent, and the idea that a top journal marks quality, are added on top. The authors admit the controls are 'proxies' and the ground truth 'unknown'. Every outcome is absorbed: a flagged top-journal paper shows the problem is spreading, and a missed fake is a limit of the tool. The labels that define the groups are the same labels the results are then used to confirm.
From the study
Throughout this paper, controls should be regarded as proxies for quality research, not verified genuine papers.
control papers were selected from high impact factor journals (decile 1) and countries under-represented in the Retraction Watch database
We assumed that all papers in the paper mill datasets are indeed paper mill products, and that the controls are legitimate scientific papers. However, we acknowledge that the ground truth is unknown
False negatives are disproportionately associated with first authors affiliated in China as they account for 90% of false negatives (compared with 47% in the overall validation dataset).