Qwen3.5-2B distilled for math-content extraction from web pages

Give it the raw text of a web page; it returns just the mathematical content, verbatim.

This revision fixes the long-page blind spot that the previous release documented as its sharpest limitation. The corpus this model was originally distilled on had been normalised with an upper cut that dropped pages over 20,000 characters rather than truncating them — so the model had never seen a page longer than 19,997 chars, while 28.1% of a realistic target corpus is longer than that. This revision continues training on 22,256 pairs built from the pages that cut had been discarding.

This is a student of GPT-5.6, not an oracle. Every number below measures agreement with the teacher's extraction on held-out pages. It does not measure whether the teacher was right.

What changed: long pages

Measured on 373 held-out pages, every one longer than 20,000 characters (median 34,383, max 49,965) — the band the previous model had never been trained on. Content-disjoint from training, verified by content hash and explicit id.

Paired bootstrap over the same 373 pages: ΔcharF1 = +0.0171, 95% CI [+0.0063, +0.0280] — significant. Δcoverage = −0.0092, 95% CI [−0.0182, +0.0001] — not significant, but the upper bound sits on zero and the coverage < 0.9 share rose by half. The gain is real and the coverage cost is real. This revision extracts long pages more accurately and slightly more conservatively.

How to read the order-preservation column

Feeding the gold extraction itself through this metric scores 90.41%, not 100%. Both arms above exceed that. So order-preservation is not bounded by the teacher's own score, and a model that emits less finds it easier to keep order. Read 92.65 together with the coverage drop, not as an independent win. (Gold-as-prediction also gives charF1 1.0 and coverage 0.9539 on this ruler — use those as reference points, not ceilings.)

What did not change: short pages

The risk of training on long pages is forgetting the short ones. Measured on the original 498-page held-out slice:

ΔcharF1 = +0.0000, 95% CI [−0.0045, +0.0043] — no forgetting.

The third column is why the training mix is 25% short pages. An earlier control trained on long pages only lost ground on every moved metric: charF1 −0.0011, order −1.11 points, repeat rate +31%. Without that control, "no forgetting" would have been indistinguishable from luck.

More data would not have helped — but removing the length cap did

Going from 100,000 to 197,269 rows moved charF1 by +0.0007, 95% CI [−0.0037, +0.0054]. Fitting y = C − A·n^(−b) gives a ceiling of 0.9512; another +0.005 would take ≈4.6M pairs, 23× the training set.

Against that backdrop, removing the 20,000-character normalisation cap bought +0.0171 on long pages in a single 3.2-hour run — roughly 24× what doubling the training data bought. The bottleneck was never corpus size; it was a normalisation upper bound that silently discarded 1.79M long pages.

Usage

The prompt is not optional — the model was trained with this exact instruction as a prefix, and it is shipped as extract_prompt.txt.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

M = "path/to/this/model"
tk = AutoTokenizer.from_pretrained(M)
model = AutoModelForCausalLM.from_pretrained(M, dtype=torch.bfloat16, device_map="cuda")
PROMPT = open(f"{M}/extract_prompt.txt").read().rstrip()

def extract(page_text: str, max_new_tokens: int = 24576) -> str:
    ids = tk(PROMPT + "\n\n" + page_text + "\n\n", return_tensors="pt").to(model.device)
    out = model.generate(**ids, max_new_tokens=max_new_tokens, do_sample=False)
    return tk.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)

Greedy decoding. Size max_new_tokens to the page, not to a constant. Our long-page evaluation used min(24576, 1.25 × input_tokens + 256); a flat 8192 would have truncated the gold answer on 25.2% of the long held-out set, and a flat 6144 on 49.3%. Extraction output scales with input, so a constant cap silently penalises exactly the pages this revision fixed.

vLLM 0.28.0 serves this model (Qwen3_5ForCausalLM, hybrid linear attention with a native Triton backend). On CUDA 12.x drivers install the cu129 build rather than the default PyPI wheel, which is built against CUDA 13.

Training recipe

Continued training from the 197,269-pair checkpoint on 22,256 additional pairs: 15,000 pages ≥20,000 chars (up to 49,998), the 1,692 pairs that the previous run had truncated, and 5,564 short pages mixed back in to prevent forgetting.

One epoch, lr 1e-5, cosine schedule, warmup ratio 0.03, weight decay 0, grad-norm clip 1.0, bf16, SDPA attention, gradient checkpointing, seed 7. Per-device batch 1 × grad-accum 4 × 4 GPUs = effective batch 16, held identical to every earlier rung. 191 minutes on 4×A800-80GB.

Max sequence length 57,344 — zero truncation. The longest pair in the mix is 57,004 tokens, measured exactly over all 22,256 rows rather than estimated from a sample (a 400-row sample put the maximum at 51,379; sampling estimates medians and quantiles, not maxima, and setting the limit from a sampled maximum would have silently clipped the tail).

Loss is masked to the output span only. To fit a 248,320-token vocabulary at 57K sequence length, cross-entropy is computed only at the supervised positions and in chunks, with the logits recomputed under torch.utils.checkpoint during backward — mathematically identical to the dense loss (verified: identical loss to 1e-7, gradient cosine > 0.9999999), with a constant 1.89 GiB logits peak instead of 25.9 GiB.

Use per-device batch 1 with variable-length data on this architecture. Padding a micro-batch corrupts the convolutional/recurrent state of the linear-attention layers; at batch 2 we observed a loss spike at step 80 and NaN weights by step 90, on data that trains cleanly at batch 1.

Limits — read before running it over a corpus

• The 20,000-character blind spot is fixed, not eliminated. Long-page charF1 is 0.8668 against 0.9454 on short pages. Long pages remain measurably harder; the gap narrowed, it did not close.
• Coverage regressed slightly on long pages (0.9781 → 0.9689; coverage < 0.9 5.36% → 8.04%). If your use case penalises missing content more than paraphrase, evaluate this directly before adopting the revision.
• Formula-token excess rose on long pages (18.50% → 20.91%). The criterion counts any excess of one token and the prompt itself mandates notation repair, so this is not 20.91% hallucination — but the direction is real. Measure repetition directly (we use a 20-gram repeat rate, which improved) rather than trusting the token count as a proxy.
• Empty output is a valid answer. The prompt instructs the model to emit nothing when a page has no substantive math. The 0.00% empty rate is on math-bearing pages and says nothing about behaviour on non-math pages.
• Beyond ~50,000 characters is still untested. The training mix tops out at 49,998 chars and the long ruler at 49,965. max_position_embeddings is 262,144, but that is the base model's allowance, not evidence.
• Faithfulness is measured against the teacher, by construction. This student was not audited against source pages directly.
• English web pages of mathematical content. Other languages and other domains are untested.

Evaluation data

• Short ruler: 498-page stratified slice of the dataset's own heldout split, content-disjoint from training.
• Long ruler: 373 held-out pages, 100% over 20,000 characters, median 34,383. Intersection with the training mix is 0 by normalised-content hash and by explicit id.
• Both arms of every comparison were scored by the same code path on the same rows; only the checkpoint differed.