Data: sourcing, filtering, dedup, decontamination

1

Concept

At scale, “the dataset” is not one file. It is a sequence of decisions: where documents came from, whether collection was authorized, what was discarded, how duplicates were detected, how languages were balanced, and which version reached a training run. Those choices shape model behavior as directly as architecture does.

Sourcing begins with provenance. Each document needs a stable identifier and metadata for source, acquisition date, language, licence or permission basis, and collection method. Publicly reachable does not automatically mean legally or ethically suitable for every use. Robots policies, contractual terms, personal data, copyrighted material, and removal requests require explicit governance. A URL alone is weak provenance because content changes.

Filtering removes material according to declared policy. Mechanical checks catch empty files, parser failures, extreme repetition, binary garbage, and implausible character distributions. Language identification supports routing and mixture control. Quality classifiers can rank documents, but they inherit the values and biases of their labels. A filter that prefers one prestige dialect may erase useful regional language. Record scores and thresholds so choices can be audited and revised.

Safety filtering is not a single keyword list. Personal identifiers, secrets, malware, sexual exploitation material, and other prohibited content require layered controls and access restrictions. Automated detectors have false positives and false negatives. High-risk review must protect the reviewers as well as the corpus. Data minimization is stronger than promising that the model will forget sensitive text later.

Deduplication reduces repeated training examples. Exact hashes catch byte-identical documents after canonicalization. Near-duplicate methods compare shingles, fingerprints, or locality-sensitive hashes to find copied pages with small edits. Decisions can happen within a source, across sources, or across train and validation splits. The unit matters: removing an entire book because one standard licence paragraph repeats would be excessive.

Duplicates distort sampling weights. If one article appears on a thousand scraped mirrors, the optimizer effectively sees it a thousand times. That can encourage memorization, reduce diversity, and create leakage between splits. Research has found benefits from deduplicating language-model training data, but the precise effect depends on corpus and method. Keep cluster membership and the representative-selection rule.

Decontamination protects evaluation. It searches candidate training data for benchmark questions, answers, code tests, or close variants and excludes or flags overlaps. Exact string matching misses paraphrases; fuzzy matching creates false positives on common phrases. The threshold, normalization, benchmark version, and decision record are part of the reported result. A score without contamination analysis can overstate generalization.

After filtering, mixture construction chooses how much data to draw from each domain and language. Raw web volume should not decide this accidentally. High-quality but small domains may be upsampled; enormous noisy sources may be capped. Upsampling also increases repetition, so token budgets and effective epochs must be tracked per source.

Version every artifact: raw snapshot, normalized text, filter configuration, duplicate clusters, decontamination report, mixture manifest, and tokenizer. Preserve counts before and after each stage without retaining unnecessary sensitive samples. A reproducible manifest should let another authorized operator reconstruct which document IDs were eligible for a run.

The central lesson is that data quality is not a scalar. Coverage, freshness, legality, language balance, safety, duplication, and evaluation independence can conflict. A serious pipeline makes those tradeoffs visible. The model will learn the distribution you actually feed it, not the dataset description in a launch post.