How to read an ML paper

1

Concept

An ML paper is an argument supported by a designed experiment, not a container of unquestionable facts. Your job is to recover five things: the problem, the proposed mechanism, the evidence, the boundary of the claim, and the information needed to reproduce or use it. Reading linearly from the first sentence to the last often hides that structure under notation and related work.

Use a first pass for triage. Read the title, abstract, introduction, section headings, figure captions, tables, conclusion, and limitations. Write one sentence each for the question, contribution, and strongest evidence. Note the publication date and venue, but do not use prestige as a substitute for inspection. If you cannot state what changed relative to the baseline, you do not yet understand the claim.

On the second pass, reconstruct the experiment. What data was used, and could it overlap with evaluation? Which baselines were compared, and were they given similar tuning and compute? What metric was optimized, and does it measure the property named in the prose? Look for sample counts, variance, confidence intervals, multiple runs, and failure cases. A table’s bold number may be numerically larger yet practically indistinguishable or purchased with much more compute.

Separate method claims from empirical claims. An equation may define an algorithm exactly. It does not establish that the algorithm is better in the world. Conversely, a benchmark result demonstrates performance only under that dataset, protocol, model scale, and implementation. Words such as “general,” “efficient,” or “robust” require operational definitions. Efficient in training FLOPs may be slow in wall-clock inference; robust to one perturbation may fail on another.

Ablations remove or alter components to test what caused an improvement. They are useful when they isolate one factor while keeping others comparable. Check for interactions: component A may matter only with component B. Scaling plots should show enough points to support the curve being claimed. Qualitative examples should be treated as illustrations, not estimated failure rates, especially when authors may select favorable outputs.

Use the third pass selectively. Re-derive the equation central to your task, trace tensor shapes, inspect appendices, and compare released code with the described method. Read dataset licenses, preprocessing, prompts, decoding settings, and hardware details. Reproduction may mean matching the exact number, reproducing the trend, or independently testing the hypothesis; state which one you attempted. Missing artifacts are evidence about reproducibility, not proof that the scientific claim is false.

Citations need their own discipline. A paper citing another source does not mean the source supports the sentence you care about. Follow the citation to the primary work and inspect its actual scope. Version dates matter because arXiv revisions, conference versions, and repository code can differ. When two public numbers do not reconcile, report the discrepancy and possible causes instead of inventing a clean average.

Finally, convert reading into a decision record. Write the claim you accept, the conditions under which you accept it, the strongest caveat, and what would change your mind. For implementation, add shapes, pseudocode only if clearly labeled, and runnable tests from the actual contract. For research, list unresolved confounders. A good reading does not require agreeing with the authors; it requires being able to explain precisely what their evidence does and does not establish.