You find a paper late at night. The abstract is confident, the headline number sits a few points above everything else, and someone has already dropped the link into a group chat with the word "important." The temptation is to skim, nod, and pass it along.
Resist that. Ten focused minutes of skepticism will tell you whether a result deserves space in your notes, your slides, or your team's roadmap — or whether it collapses the moment you ask what it was actually compared against.
None of this assumes bad faith. Most research is done carefully by people who want to be right. The trouble is narrower: what a paper proves and what a paper suggests are different things, and the gap widens with every retelling.
The checklist at a glance
Here is the whole thing in compressed form. The sections after it explain how to apply each point.
- State the narrowest claim the evidence actually supports.
- Ask what would break it. If nothing could, the claim isn't testable.
- Check the dataset: provenance, splits, duplicates, contamination.
- Check the metric and whether it measures what the claim is about.
- Check the baselines: re-run or copied, tuned or default, matched compute or not.
- Look for variance: seeds, error bars, repeated runs.
- Read the ablations to see which component is doing the work.
- Read the limitations and appendix before trusting the introduction's promises.
- Look for released code and data, then try to trace one number.
- Write the caveat you'd attach if you shared or cited the paper.
Methodology: ask what would break the result
A method section is a story about cause and effect. Your job is to find the steps where the story gets thin.
Start with the ablation table. If removing a component costs two points, that component matters. If removing it costs nothing, the framing is doing more work than the technique. Watch for ablations that remove three things at once — you learn that the bundle matters, not which part of it.
Then check the budget. A model trained with far more compute, data, or tuning than its competitors is not a clean test of an idea, even when the table looks tidy. Careful papers report matched-compute or matched-parameter comparisons; weaker ones hope you won't ask.
One blunt question finishes the job: if I changed a single design decision, would the headline result survive? If the answer is "probably not," you're looking at a fragile result — worth knowing, but not worth building on.
Datasets and evaluation: where most surprises hide
Dataset problems sink more claims than modeling problems do, and they're harder to spot from the abstract.
Look at provenance first. Where did the data come from, how was it filtered, and who labeled it? Then look at the splits. Duplicate or near-duplicate examples across train and test inflate scores without anyone intending fraud. Public benchmarks reused for years can end up inside the training data of large models, which turns a leaderboard into a memory test.
Preprocessing deserves a second look. Different tokenizers, image resizing, prompt formats, or answer-normalization rules can produce apparent gains that disappear when both systems are treated the same way.
Finally, check the metric against the claim. Accuracy on a balanced set says little about rare cases. A single similarity score says little about whether outputs are useful. If the paper claims robustness, look for evaluation under distribution shift rather than a second clean test set.
Baselines and tables: reading numbers like a skeptic
Weak baselines are the most common flaw in otherwise solid work. Three quick tests:
- Were baselines re-run? Numbers copied from an earlier paper may come from different splits, budgets, or preprocessing.
- Were they tuned? A baseline left at default settings, compared against a carefully tuned proposed method, tells you almost nothing.
- Is the gain bigger than the noise? A 0.3-point improvement with one seed and no error bars is a coin flip wearing a suit.
When a paper reports several runs, read the spread, not just the mean. When it reports one, treat small differences as ties — and say so when you cite it.
Claims, limitations, and the gap between them
Jump to the limitations section before you read the introduction's promises. Authors who name their own weaknesses tend to be more reliable than those who don't have any.
Then watch the verbs. "Outperforms on benchmark Z" is specific. "Achieves stronger reasoning" is a leap. Note what was never tested: other languages, longer inputs, noisier inputs, smaller models, real users.
Check where the paper lives, too. A preprint has not been through peer review — that doesn't make it wrong, but it means the scrutiny is yours to supply. A paper at a selective venue has survived review, which is evidence of plausibility, not correctness. Citation counts measure attention, not truth.
Turning the checklist into a habit
Keep a three-line note for every paper you care about: the claim, the strongest evidence for it, and what's missing. That's it. Fifteen minutes, once, and you'll never again have to reconstruct why you trusted something six months ago.
When you share or cite, share the caveat alongside the link. If you can't state that caveat in one sentence, you probably haven't finished reading. And when a paper genuinely matters to a decision, reproduce one number from it: rerun a baseline, evaluate on your own data, check the released code. One honest reproduction beats a hundred enthusiastic retellings.
