e2etext-T1edu1-T1
A single Qwen3-0.6B model that turns raw HTML into clean pretraining text, and the complete pipeline that produced the released 7,436,788,128-token corpus of the same name.
Instead of the usual two-stage scraper + rule stack, one Qwen3-0.6B student reads a canonical, line-numbered rendering of the whole page and emits one of four decisions — keep, edit, delete, rewrite — followed by the payload that decision implies. A deterministic executor turns that prediction into the corpus text.
This repository holds three things:
- the student checkpoint that produced the corpus (
model.safetensorsand its tokenizer files, at the root) together with the system prompt it was trained and run under (system_prompt.txt); - the documentation of the method end to end, with every number traced to the file, line or log it came from;
- byte-identical copies of the scripts that ran.
Where the artifacts live
| Artifact | Location |
|---|---|
| Student checkpoint (11 files, 1,208,026,637 B) | this repository, root |
| System prompt | system_prompt.txt here; identical copies at scripts/prompt_sft_system_rwmix_t1_rewrite_all_edu1_rw30_ep3.txt and in the dataset repo under exp0918_artifacts/prompts/ |
| Tokenized corpus, 444 shards | dataset repo jerryyan/dclm-pool-400m-1x-student-cleaned-rescued, folder students/e2etext-t1edu1-t1/ |
| Continued SFT set (131,484 rows) | dataset repo, exp0918_artifacts/sft_data/sft_rwmix_t1_rewrite_all_edu1_rw30_ep3.jsonl, with its own README next to it |
| Rendered model inputs for the whole pool | dataset repo, exp0918_artifacts/inference_input/rendered_text/ |
| 1,000-row SFT sample and five executed worked examples | samples/ here |
Using the model
Input format
The model expects a canonical, line-numbered rendering of the page, not raw HTML and not arbitrary text. The renderer is part of the method (docs/01_rendering.md):
- drop
head,style,script,noscript,link,meta,iframe,frame - render with the WebKit-based
webpage_convertertext renderer shipped in Dripper (MinerU-HTML 1.0.0) - strip residual markup with BeautifulSoup 4.15.0
get_text(separator=" ") - collapse runs of
|, blank separator-only lines, intra-line whitespace; unescape Markdown punctuation; drop empty lines - prefix each remaining line with
<lid:N>
Using a different renderer shifts the input distribution away from training. The system
prompt matters and ships as system_prompt.txt; read the limitation about it below.
Output format
<keep>|<edit>|<delete>|<rewrite> <- line 1: the decision
<extract> <- line 2, always literal
rm A / rm A-B ... <- the extractor's whole-line removals
<same decision again> <- stage boundary (a format artifact, not a 2nd prediction)
<payload> <- edit ops, or replacement text, or nothing
rm N / rm A-B remove whole lines. sub N: "s" removes the exact quoted substring from
line N — it never substitutes replacement text. A deterministic executor
(scripts/e2e_ops.py, body_from_prediction_dfirst) applies the union of both stages and
records the path taken as e2e_tag (<keep>→extract, <edit>→refine,
<delete>→delete, <rewrite>→rewrite).
A worked example — the student's whole output for one page:
<edit>
<extract>
rm 1-39
rm 71-111
<edit>
sub 41: "Apply now "
rm 70
Five worked triples — line-numbered input, serialized target, executed corpus text — covering all four decisions and the mixed whole-line + substring edit are in samples/WORKED_EXAMPLES.md. They are produced by running the shipped executor, not written by hand.
Decoding
The released corpus used temperature 1.0, top-p 1.0, max_new_tokens=3072,
max_model_len=32768, thinking mode disabled (vLLM 0.11.1). That is what the -T1 suffix
means. Anyone running the model directly should apply the same post-filter the corpus went
through (docs/05_inference.md): drop outputs with two or more
operation-like lines or a short bare-tag prefix, an out-of-vocabulary tag, or a standalone tag
line. Measured on this corpus that removes 0.0614% of documents.
The six stages
| # | Stage | What happens | Doc |
|---|---|---|---|
| 1 | Canonical rendering | Raw HTML → WebKit text → whitespace/pipe/Markdown cleanup → <lid:n> line numbering. The same renderer builds the SFT inputs and the full-corpus inputs, so there is no train/inference rendering shift. | docs/01_rendering.md |
| 2 | Teacher labels | Dripper extracts, Qwen3.8-27B prunes or deletes under a strict-subset prompt, FineWeb-Edu gates the deleted pile, and RePro's 1B rephraser (cx-cmu/repro-rephraser-1B) writes replacement text for the pages that survive the gate. | docs/02_teachers.md |
| 3 | SFT data | Teacher outputs are compiled into a decision-first serialized target over the line ids. A 1,383,115-row base label set, then a 131,484-row continued set that raises the rewrite class to 30%. | docs/03_sft_data.md |
| 4 | Training | Qwen3-0.6B, base run then a 3-epoch continued run at global batch 192 with the four decision tokens upweighted ×5. | docs/04_training.md |
| 5 | Pool inference | The student runs over the full 10,318-shard HTML universe at temperature 1.0, the executor materializes the text, and a post-filter removes malformed generations. | docs/05_inference.md |
| 6 | Corpus | BFF Bloom deduplication (13-grams, threshold 0.8), GPT-NeoX-20B tokenization. No rule filter, no fastText filter. | docs/06_corpus.md |
Held-out teacher fidelity and how to reproduce it: docs/07_evaluation.md. Internal run tags translated into plain language: docs/NAMING.md.
Training
Continue-trained from a base whole-page student, not from Qwen3-0.6B directly (docs/04_training.md).
| Value | |
|---|---|
| Teachers | Dripper (extraction) → Qwen3.8-27B (strict-subset cleanup) → RePro cx-cmu/repro-rephraser-1B (rewrite), with FineWeb-Edu as the gate |
| Base label set | 1,383,115 rows labelled by Dripper + Qwen3.8-27B: 513,724 keep / 359,292 edit / 510,099 delete; parent of both base students |
| Base student's training set | the same 1,383,115 rows relabelled under the earlier rescue policy (1,376,248 packed sequences; <rewrite> is 0.11% of it, and the base student never emitted the tag) |
| Initialization of the continued run | the epoch-2 checkpoint of that 3-epoch base run |
| Continued SFT set | 131,484 examples: 35,189 keep / 24,611 edit / 32,239 delete / 39,445 rewrite (30.0%) |
| Rewrite selection | pages the 27B deleted whose saved Dripper text scores FineWeb-Edu ≥ 1.0 — no verbatim-keep branch, no DataMan, no acceptance gate |
| Rewrite targets | cx-cmu/repro-rephraser-1B at temperature 1.0, top-p 0.9 |
| Epochs / steps | 3 / 336 |
| Global batch | 192 (4 nodes × 8 GPUs × micro 1 × accum 6) |
| Peak LR | 8e-5, ~11-step warmup then cosine |
| Sequence length | 32,768, packed |
| Decision-token loss weight | 5× |
Headline numbers
| Quantity | Value |
|---|---|
| Released corpus | 7,436,788,128 GPT-NeoX-20B tokens, 444 webdataset shards, 19.85 GB |
| Pool | 10,318 shards of raw HTML; 10,646,236 documents written before post-filter |
| Post-filter | 6,537 documents dropped (0.0614%), 10,639,699 kept |
| Student | Qwen3-0.6B, 336 optimizer steps of continued training, global batch 192 |
| Continued SFT set | 131,484 examples — 35,189 keep / 24,611 edit / 32,239 delete / 39,445 rewrite (30.0%) |
| Base label set | 1,383,115 rows — 513,724 keep / 359,292 edit / 510,099 delete |
| Held-out decision accuracy | 76.25% on 960 pages |
| Held-out corpus-text token F1 | 86.94% (precision 84.63%, recall 89.38%) |
Every one of these is traced to a file, line or log in VERIFICATION.md, which also lists the handful of things that could not be verified.
Evaluation — teacher fidelity on 960 held-out pages
Held out by shard, disjoint from all training shards. The gold labels a teacher-deleted page
<rewrite> by the same rule as the continued SFT set (FineWeb-Edu ≥ 1.0 on the Dripper text, with
a temperature-1.0 RePro-1B rewrite as the target); see docs/07_evaluation.md, which also records
the correction of an earlier gold built with a different rewrite rule.
| Decision | Gold | Recall | Precision |
|---|---|---|---|
<extract> (keep) | 295 | 79.7% | 76.3% |
<refine> (edit) | 286 | 69.6% | 76.5% |
<delete> | 351 | 79.2% | 88.0% |
<rewrite> | 28 | 71.4% | 26.3% |
| Avg (accuracy) | 960 | 76.25% | 76.25% |
Corpus-text agreement, scored on the text that would actually enter the corpus (deleted pages target the empty string), all 960 pages:
| Metric | Value |
|---|---|
| Token precision | 84.63% |
| Token recall | 89.38% |
| Token F1 | 86.94% |
| Keep/delete binary accuracy | 88.44% |
<extract> step vs. Dripper, word-token F1 | 94.81% |
No downstream DCLM Core score is reported in this repository; see docs/07_evaluation.md for why and how to produce one.
Limitations — please read before comparing
- The training system prompt states a policy the labels contradict.
system_prompt.txtdescribes a three-band rescue (keep ≥ 1.5 verbatim, paraphrase 1.0–1.5, with an acceptance test on the paraphrase). The labels implement a single threshold at 1.0 with no verbatim-keep branch and no acceptance test. The prompt generator substitutes only the class-frequency line and inherited the rest from a template written for the other policy. This is a genuine defect, and the prompt is shipped verbatim so it can be inspected. - Decision accuracy is lower on long pages: 78.3% on the two shorter length quartiles against 73.8% on the longest.
- Malformed generations occur at roughly 0.06% of documents and are removed by the post-filter described above, not by the decoder.
- No RL, no preference tuning. Supervised fine-tuning only.
What is in this repository
README.md this file (model card + index)
model.safetensors, config.json, tokenizer.json, ... the 11-file student checkpoint
system_prompt.txt the system prompt used for training and inference
docs/ the six stages, evaluation, and the naming map
scripts/ byte-identical copies of the scripts that ran, + MANIFEST.md
samples/ the 1k SFT sample and five executed worked examples
hf/ the dataset-side cards and the publication record
VERIFICATION.md every claim -> its evidence; and what could not be verified
WORKLOG.md how this repository was assembled and published
LICENSE Apache-2.0 (code, documentation and model weights)
scripts/ contains unmodified copies. They hardcode /mnt/weka/shrd/ci/zichunyu/jerryy2
paths, cluster partitions and a conda prefix; the substitutions you need are listed in
scripts/MANIFEST.md rather than edited into the files, so that the
copies keep matching what actually ran.
Scope
This repository ships the causal path to e2etext-T1edu1-T1 and nothing else. Sibling
experiments — other rescue-selection rules, other rewrite fractions, other students — are
referenced only where they are genuine analysis, and are labelled as such. Their data and
scripts are not included.
License
The code, documentation and model weights in this repository are released under Apache-2.0 (the student is a fine-tune of Qwen3-0.6B and inherits that model's license). The corpus and the SFT set follow the DCLM pool they are built from (ODC-BY) and are documented in the dataset repository.