bonsai-qwen3-30b-a3b-teacher-corrected-lora
Qwen3-30B-A3B-Base, corrected toward TypeScript via LoRA fine-tuning (adapter merged into the base weights) rather than full-parameter fine-tuning. Part of bonsai, a project pruning and distilling this model into a smaller TypeScript-specialized one.
This checkpoint exists to answer a specific question: does the fine-tuning
method used for teacher correction matter, not just the token budget? A
full-parameter correction of this same base model
(bonsai-qwen3-30b-a3b-teacher-corrected)
improved validation loss but regressed on HumanEval-TS pass@1 relative to
the uncorrected base model (0.5660 vs. 0.6708, full 159-problem set).
2408.11796 explicitly flags "lighter
fine-tuning such as LoRA" as untested future work for this exact correction
step -- this checkpoint tests it.
Downstream benchmark results for this checkpoint are not yet available. This card will be updated once HumanEval-TS/MBPP-TS evaluation runs.
Training
- Base model:
Qwen/Qwen3-30B-A3B-Base(30.5B total params, MoE, 3B active) - Method: LoRA (rank 16, alpha 32, dropout 0.05), attention projections
only (
q_proj,k_proj,v_proj,o_proj) -- not the experts, which dominate the model's parameter count; a scope choice for tractability, not an oversight. 13,369,344 trainable parameters (0.044% of 30.5B). Adapter merged into the base weights after training (merge_and_unload), so this is a plain dense checkpoint, loadable and deployable identically to any other model in this family -- no PEFT-specific handling needed. - Data: TypeScript source from
bigcode/the-stack-dedup, one epoch, no repeats -- 6,000 samples (1500 steps x batch size 4), disjoint from the calibration pool used to score validation loss - Data postprocessing (fixed after a data-quality audit found real
issues in the pipeline the full-parameter run used): samples from
node_modules/paths or.d.tsfiles are excluded (auto-generated / vendored, not representative of hand-written TypeScript); samples over 2048 tokens are dropped rather than truncated, so no training example ends mid-statement. 7,800 samples were pulled to yield the final 6,000 that passed all filters (23.1% dropped). - Loss: plain LM cross-entropy, no distillation at this stage
- Optimizer: plain AdamW (LoRA's trainable footprint is small enough that 8-bit AdamW / memory-saving tricks weren't needed)
- LR schedule: linear warmup (15% of steps) -> cosine decay to a floor; peak LR 1e-4, floor 1e-5 -- higher than the full-parameter run's 2e-5, since LoRA updates a much smaller subspace and needs a larger step to move as far in the same number of steps
- Sequence length: up to 2048 tokens, unpadded (gradient-accumulated single-sample forward/backward)
- Hardware: 1x H100 (80GB PCIe) -- LoRA's small trainable footprint fits on a single GPU without the multi-GPU sharding or 8-bit optimizer the full-parameter run needed
- Steps: 1500
Result
| validation loss | |
|---|---|
| first recorded checkpoint (step 25) | 0.8927 |
| final (step 1500) | 0.8346 |
Validation loss measured on the same held-out calibration shard as the full-parameter run (shard-level disjoint from the retraining pool by construction). Note: this table's "first recorded" value is the training loop's first periodic validation check (25 steps in), not a pre-training baseline measured separately -- the full-parameter checkpoint's reported "before correction" figure (0.9331) used a distinct measurement, so the two runs' deltas aren't directly comparable, but the final values are (same calibration set, same loss function): this run's final 0.8346 is lower than the full-parameter run's final 0.8842.
What this is not
- Not benchmarked on HumanEval-TS/MBPP-TS yet -- that's the next step.
- Not pruned. Still the full 30.5B-parameter MoE.
- Not distilled from itself -- fine-tuned directly on TypeScript text, no separate teacher involved at this stage.
Reproducing this
Training code: src/bonsai/training/
in the bonsai repo (the LoRA wrapping and data-filtering logic around it is
a one-off orchestration script, not committed per the project's convention
of keeping reusable library code separate from single-run scripts).
License
Apache 2.0, inherited from Qwen3-30B-A3B-Base.