SymQwen3-1.7B-CDLM-SDLM-5k
Staged symmetrized arm: initialized from CDLM-5k; attention pattern constrained to rownorm(A + A^T) of the model's own causal pattern A. Must be evaluated through this sym forward (config.attn_mode is pre-set).
Part of the DLM1B qwen3_sym campaign (2026-08-26/27): a matched-total-compute comparison of AR→DLM conversion recipes for Qwen3-1.7B — direct bidirectional (BDLM), a causal mask-tolerance stage (CDLM), staged adaptations with free (BDLM) vs hard-symmetrized (SDLM) attention, and a λ-sweep of blended rules (MIX: λ·symmetrized + (1−λ)·free). Every path totals 10k steps at GBS 256 × L2048 (~5.2B tokens), lr 1e-5, wd 0, WSD 100/*/500, ADLMC v3 data mixture + frozen aug, all-position clean-target supervision.
⚠ SDLM eval rule.
config.attn_moderecords the training-time attention semantics and the loaded model applies it automatically. Do NOT override it — in particular, running the CDLM-SDLM checkpoint through free bidirectional attention collapses it instantly (established in the DLM1B toy laboratory, Exp 4). All state_dicts are architecturally identical across modes; the mode is forward semantics, not weights.
Endpoint results (all arms, for context)
| model | HE static-32 | HE τ0.95 | MBPP-499 |
|---|---|---|---|
| BDLM-10k | 27.4 | 27.4 | 25.1 |
| CDLM-5k (stage) | 9.1 | — | 10.0 |
| CDLM→BDLM-5k (λ=0) | 23.2 | 23.2 | 27.9 |
| CDLM→MIX-5k λ=0.1 | 26.2 | 26.2 | 27.7 |
| CDLM→MIX-5k λ=0.5 | 27.4 | 27.4 | 28.7 |
| CDLM→SDLM-5k (λ=1) | 26.2 | 23.2 | 26.1 |
All pass@1, greedy, generation canvas 256 (dQwen house harness; __g256
cells — NOT comparable to gen-1024 numbers elsewhere). τ0.95 = adaptive
low-confidence commit, threshold 0.95. MBPP-499 = MBPP test minus task_id
493 (its 4.3k-char prompt starves a 2048 canvas). ⚠ Single seed, single
greedy decode: per-problem paired tests (McNemar) find NO statistically
significant difference between any two models; the consistent-sign finding
is that only the hard-mirror (λ=1) arm degrades under adaptive commit.
λ-comparisons are valid only within the staged family (same init/data/
steps); BDLM-10k has 2× the bidirectional-mode steps and 2× unique tokens.
Headlines: (1) conversion quality is remarkably ROBUST to the bidirectional mixing rule — hard mirror, free, and every tested blend land within ~2.5 points, none pairwise-significant; (2) the one consistent-sign effect is calibration under adaptive decode — only the hard-mirror (λ=1) arm loses answers at τ0.95, any free component makes adaptive commit a pure ~25% speedup; (3) λ=0.5 posts the best aggregates (interior maximum; hypothesis-grade, n=1). Details in the DLM1B campaign notes.
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"EER6/SymQwen3-1.7B-CDLM-SDLM-5k", trust_remote_code=True, dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("EER6/SymQwen3-1.7B-CDLM-SDLM-5k")
Forward contract (from ADLMC's DiffuQwen3): model(input_ids) prepends BOS
internally; logits has shape (B, L, vocab) aligned to the input positions.
Special ids: MASK = 151660 (<|fim_middle|>), PAD = 151643, BOS = 151652.
The sym forward assumes a full canvas (attention_mask None/all-ones) and uses
no KV cache.
Provenance
- Training: Vista (TACC) job 943816, 5k (from CDLM-5k) steps, attn_mode
sym; full step-level log intraining_log.jsonl, exact CLI args inargparse.json. - Code:
DLM1B/qwen3_sym@ a431778 (self-contained stack; validation gates inverify_port.py— aug byte-parity, bitwise loss parity on real weights, save→reload).