Tritfold 1.7B ternary · feature-distilled (PTQ1_0, 1.75 bpw) — v0.6
📓 Code, notebooks, all findings & full experiment log: github.com/benzeng/tritfold
The first open ternary model with both knowledge ranking AND generation capability — targeted in-distribution corpus through teacher KL contributes most of the knowledge gain (+0.060 sciq), hidden-state cosine matching adds a further +0.027 acc_norm / +0.065 raw acc — see the control-experiment note below.
What makes v0.6 different
| v0.1-v0.3 (KL only) | v0.6 (KL + cosine + ultra) | |
|---|---|---|
| sciq acc_norm | ~0.40 (57% FP) | 0.485 (72% FP) |
| ARC acc_norm | ~0.26 (69% FP) | 0.279 (74% FP) |
| Generation | ❌ loops / confabulation | ✅ coherent text restored |
| wiki ppl | 1.24-1.41×FP | 1.03×FP |
| Size | 424 MB | 424 MB (same) |
The "knowing vs saying" discovery
The core finding: distribution matching and knowledge internalization are independent dimensions.
- Output-level KL → transfers distribution shape (how to speak) → bounded by teacher quality
- Targeted in-distribution corpus through teacher KL → transfers discrimination ability (+0.060 sciq over generic-corpus mixes); generic educational corpus transfers none (+0.004)
- Feature-level cosine (OFF) → adds a modest increment on top: +0.027 acc_norm (within noise) / +0.065 raw accuracy (2.7σ) — sharpens argmax selection
- Generation fluency → requires instruction data (ultrachat) in the mix
- Factual recall → still capacity-bound (1.58 bits can't store all facts)
In practice: the model can select "Mars" from options (sciq 72% FP) but only sometimes generate "Mars" in free text. RAG complements this perfectly.
Post-release control experiment (erratum)
The original card attributed the sciq jump (0.398 → 0.485) to feature-level cosine matching. A control arm (KL-only; identical data mix, steps, teacher and starting checkpoint) corrects the attribution:
| sciq | acc | acc_norm |
|---|---|---|
| v0.3 baseline (KL, generic corpus) | 0.376* | 0.398 |
| 1a control (KL + full sciq stream) | 0.471 | 0.458 |
| 1b = this model (+ cosine) | 0.536 | 0.485 |
*v0.3 measured on the full n=1000 validation; 1a/1b on the decontaminated n=845 subset — the harder set, so +0.060 is conservative.
Most of the gain came from a data-pipeline fix: earlier mixes used a per-example ≥512-token filter that silently kept only 81 of 11,679 sciq training examples (3.3% coverage, each surviving window recycled 37×); this release trains on the full 1.27M-token sciq stream. The same bug retroactively retracts the "knowledge saturation is corpus-insensitive" claim from the v0.3/v0.4 era — targeted corpus through distillation is a real knowledge pathway. Cosine matching's net contribution: +0.027 acc_norm (within noise) / +0.065 raw accuracy (2.7σ, argmax sharpening).
The artifact and all measured numbers above are unchanged. Details in the tritfold repo: findings-11 §0 and the findings-9 erratum.
Performance
| value | |
|---|---|
| sciq val (decontaminated, n=845) | acc 0.536 ± acc_norm 0.485 |
| ARC-Challenge | acc 0.246 ± acc_norm 0.279 |
| Wiki ppl (bf16 protocol) | best 26.52 (1.03× of 1.7B FP) |
| Size | 424 MB (exact 1.75 bpw, 9.1× smaller) |
| Training | ~3 A100 sessions (v0.3 → feature-distill → ultrachat recovery) |
Requirements
- The PrismML llama.cpp fork (
prismbranch) — mainline cannot load PTQ1_0. - Mandatory sampling recipe:
--temp 0.5 --top-p 0.85 --top-k 20 --repeat-penalty 1.1
License & attribution
Apache-2.0. Independent research reproduction; not affiliated with PrismML or Caltech. Derivative of Qwen3-1.7B (Apache-2.0, Alibaba Qwen team).