qwen2-style-50k
A Qwen2-0.5B-architecture decoder LM (246M parameters) trained from scratch on CCI3-HQ Chinese text with the zh-pythia 50k tokenizer, using EleutherAI's gpt-neox. It is the architecture-controlled twin of the zh-pythia GPT-NeoX-style models: same data, tokenizer, and token budget — only the transformer architecture differs (Qwen2-style GQA/SwiGLU/RMSNorm vs GPT-NeoX-style). The goal is to test how much of the Chinese-benchmark behaviour difference between the two families is explained by architecture alone.
Training is complete: 24,414 steps = 102.4B tokens (~1 epoch of the 110.3B-token
CCI3-HQ corpus, seq len 4096, ~4.19M tokens/step). Final training loss 3.02;
held-out test loss 2.707 (perplexity 15.0). main holds the final checkpoint
(step 24414); other snapshots are step<N> branches, Pythia-style: the early
ladder 1–128 plus 5000, 7500, 10000, 14500, 15000, 17500, 20000.
Load
from transformers import AutoTokenizer, LlamaForCausalLM
tok = AutoTokenizer.from_pretrained("hai-hu/qwen2-style-50k")
model = LlamaForCausalLM.from_pretrained("hai-hu/qwen2-style-50k")
# earlier steps: add revision="step<N>"
Important: checkpoint format is Llama, not Qwen2 — on purpose
The model was trained in gpt-neox with biases on all four attention
projections (Q/K/V and the output projection). HuggingFace's
Qwen2ForCausalLM hard-codes o_proj bias=False, so a Qwen2-class export would
have to drop the trained o-projection bias. On a validation batch at the final
checkpoint that bias moves logits by up to 25 (mean 3.0) and flips ~78% of
argmaxes — i.e. it is not negligible, so exporting as Qwen2 would silently
produce a different, broken model.
LlamaForCausalLM with attention_bias=true (biases on q/k/v and o_proj)
represents the trained model exactly; RMSNorm, SwiGLU, rotary embeddings,
GQA and tied embeddings are otherwise identical between the two classes.
Conversion was verified to logits parity against the original gpt-neox
checkpoint: max |Δlogit| = 4e-5 (fp32 numerics), 100% argmax agreement.
So: load it with LlamaForCausalLM (as above) and treat it as a Qwen2-family
architecture. The weights are untouched originals.
Architecture
| layers | 24 |
| hidden size | 896 |
| attention heads | 14 (head_dim 64) |
| KV heads (GQA) | 2 |
| intermediate (SwiGLU) | 2432 |
| context length | 4096 |
| positional encoding | RoPE, theta 10000, 100% |
| norm | RMSNorm, eps 1e-6 |
| embeddings | tied, vocab 50,048 padded (tokenizer: 50,002 = 50k BPE + `< |
| parameters | 246M (291M counting the tied head once per role) |
Training hyperparameters
Adam(0.9, 0.95), lr 3e-4 → cosine → 3e-5, warmup 8.2% (2000 steps), weight decay 0.1, grad clip 1.0, bf16, ZeRO-1, global batch 1024 sequences × 4096 tokens. Data: CCI3-HQ (~110B-token corpus, ~1 epoch for 24414 steps). Checkpoint every 250 steps; per-step branches here are a subset.
Hardware: 2× NVIDIA RTX PRO 6000 Blackwell (96GB), ~37.4 s/step, ~12.5 days wall clock.