Data: sourcing, filtering, dedup, decontamination

01 · Concept

Concept

Here is a question you cannot answer, and the fact that you cannot answer it is the lesson. What was Qwen3.8-27B trained on? The model card gives architecture, licence, sampling presets, and a benchmark table. It does not give a corpus, a token count, a language balance, or a cutoff date, and there is no accompanying technical report. Someone made every decision below for this model, and none of them were disclosed. That is now the norm for open-weight releases: you receive the weights and the shapes, not the pipeline that produced them.

So this lesson teaches the pipeline generically, and you carry the shape of it as a set of questions you know to ask of any release — including this one, where the honest answer to most of them is “not published.”

Sourcing begins with provenance. Each document needs a stable identifier and metadata for source, acquisition date, language, licence or permission basis, and collection method. Publicly reachable does not automatically mean legally or ethically suitable for every use. Robots policies, contractual terms, personal data, copyrighted material, and removal requests require explicit governance. A URL alone is weak provenance, because the content behind it changes.

Filtering removes material according to a declared policy. Mechanical checks catch empty files, parser failures, extreme repetition, binary garbage, and implausible character distributions. Language identification supports routing and mixture control. Quality classifiers can rank documents, but they inherit the values and biases of their labels; a filter that prefers one prestige dialect quietly erases useful regional language. Safety filtering is not one keyword list — personal identifiers, secrets, malware, and exploitation material need layered controls, access restrictions, and protection for the humans who review the hard cases. Record scores and thresholds so the choices can be audited later.

Deduplication reduces repeated training examples. Exact hashes catch byte-identical documents after canonicalization; near-duplicate methods compare shingles, fingerprints, or locality-sensitive hashes to find copied pages with small edits. The unit matters: discarding an entire book because one standard licence paragraph repeats would be absurd. Duplicates distort sampling weights, and under the next-token objective of lesson 5.1 that distortion is direct — a document mirrored a thousand times contributes a thousand times the gradient, encouraging memorization and shrinking effective diversity.

Decontamination protects evaluation. It searches candidate training data for benchmark questions, answers, code tests, or close variants and excludes or flags the overlaps. Exact matching misses paraphrases; fuzzy matching produces false positives on common phrases. The threshold, normalization, benchmark version, and decision record are part of the reported result. This is why the model card’s vendor-reported numbers — SWE-bench Pro 61.7, GPQA Diamond 89.2, and the rest — are vendor-reported and await independent reproduction: without a contamination report, a score is a claim, not a measurement.

Now the arithmetic, because mixture construction is where a pipeline becomes numeric. Suppose — and this is an illustrative assumption of ours, not a fact about any shipped model — you plan a 3-trillion-token budget and want a curated code domain to be 10% of it, but that domain holds only 50 billion unique tokens. You will draw

over that source. Meanwhile a 2-trillion-token web pool assigned 40% of the budget is drawn at $0.40\times 3{,}000 = 1{,}200$ B tokens, which is $1{,}200/2{,}000 = 0.6$ epochs — it is never seen once in full. Same corpus, same run, one source repeated six times and another sampled at 60% coverage. Effective epochs per source, not the headline token total, is the number that predicts memorization risk. Recompute it whenever a mixture weight moves.

The classic wrong turn here is deduplicating within each source and declaring the job done. Sources overlap: the same Wikipedia article reaches you through the encyclopedia dump, three mirror sites, two scraped forums that quote it, and a book that reproduces it. Per-source dedup finds none of these, because each copy is unique inside its own shard. Cross-source deduplication over the merged candidate pool is what actually collapses the cluster, and it must run before the train/validation split, not after — otherwise copies of one text land on both sides and validation loss reports a number better than any genuine out-of-source performance.

Version every artifact: raw snapshot, normalized text, filter configuration, duplicate clusters, decontamination report, mixture manifest, and tokenizer. Preserve counts before and after each stage without retaining unnecessary sensitive samples. A reproducible manifest should let another authorized operator reconstruct exactly which document IDs were eligible for a run.

The model will learn the distribution you actually feed it, not the description in a launch post. And when there is no description at all, the only responsible position is to say so.