Bala v1 (30B-A3B)
Bala v1 is a coding-and-reasoning model developed by TushLab, the model lab of AlreadyAI. It is produced by attention-only QLoRA fine-tuning of Qwen3-30B-A3B-Thinking-2507 (MoE, ~30B total / ~3B active parameters) on a small mixture of open reasoning-distillation corpora. It is an honest proof-of-capability release: a real, decontaminated improvement in mathematics over its base, with no claim of a coding gain and no frontier claim.
- Repository:
AlreadyAI/Bala-30B-A3B· Release:v1.0 - Developed by: TushLab (AlreadyAI)
Transparency: Bala is a fine-tune of the Apache-2.0 model
Qwen/Qwen3-30B-A3B-Thinking-2507. It is trained to self-identify as "Bala, made by TushLab"; the underlying base is disclosed here and in the accompanying report.
Files & usage
This repo ships both the fully-merged model (at the repo root — load
directly) and the standalone LoRA adapters (adapters/).
adapters/checkpoint-2556 is the released epoch-3 winner; checkpoint-1700 is
the epoch-2 checkpoint. The merged weights and (adapter + base) are equivalent.
Direct (merged weights):
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("AlreadyAI/Bala-30B-A3B")
model = AutoModelForCausalLM.from_pretrained("AlreadyAI/Bala-30B-A3B",
torch_dtype="auto", device_map="auto")
Adapter on the base (equivalent):
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "Qwen/Qwen3-30B-A3B-Thinking-2507"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "AlreadyAI/Bala-30B-A3B",
subfolder="adapters/checkpoint-2556")
What it is / isn't
- Is: a math-reasoning booster. On MATH-500, pass@1 improves 53.6 → 77.2 (+23.6) over the base.
- Isn't: a coding improvement (HumanEval+ −3.7, MBPP+ +3.4, GSM8K flat near ceiling), and not competitive with closed frontier models. See Evaluation.
Evaluation (greedy pass@1, full sets)
| Benchmark | n | Base | Bala | Δ |
|---|---|---|---|---|
| GSM8K | 1,319 | 95.6 | 94.2 | −1.4 |
| MATH-500 | 500 | 53.6 | 77.2 | +23.6 |
| HumanEval+ | 164 | 78.7 | 75.0 | −3.7 |
| MBPP+ | 378 | 86.0 | 89.4 | +3.4 |
| Overall | 2,361 | 84.0 | 88.5 | +4.5 |
Base = unmodified Qwen3-30B-A3B-Thinking-2507, evaluated identically. Full
methodology, the 4-config hyperparameter comparison, and eval logs are in the
technical report and results/.
Decontamination
We ran n-gram containment + exact/substring matching between the entire data
pool and all four benchmarks. The data this model was trained on
(offline corpora + identity) is clean against all four benchmarks at n=13 and
the stricter n=8. (An auxiliary pool of self-generated traces, not used to
train this model, contained ~12% MBPP+ overlap; it is disclosed in the paper and
is not part of this model's training data.) Report:
results/decontamination/report.json. Method: n-gram/exact only — does not
catch deep paraphrase.
Training
- Method: QLoRA, 4-bit NF4 base / bf16 compute, LoRA on
q,k,v,o_projonly (MoE expert MLPs frozen). Config B: rank 128, α 256, LR 2.5e-4, cosine, warmup 0.03, 3 epochs, seq-len 4096, effective batch 16, seed 0. - Data: 15,296 examples — OpenCodeReasoning (3k), Mixture-of-Thoughts code/math/science (3k each), OpenThoughts3 (3k), identity (296). Offline traces treated as verified-by-construction (not re-executed).
- Hardware: 1× AWS g6e.12xlarge (4× L40S 48 GB, no NVLink), single-GPU per config.
Intended use & limitations
Research and experimentation on math/reasoning tasks. Single-seed results (no variance reported); coding is not improved over base; identity data is mixed in (model presents as "Bala"). Not for high-stakes use without independent evaluation. Inherits the base model's licenses, biases, and context behavior.
License
Apache-2.0 (base and all released artifacts).
Paper
Technical report: "Distilling Open Reasoning Corpora into a 3B-Active MoE on
Commodity GPUs: What Transfers, What Doesn't, and a Decontamination Check" —
published on Zenodo, DOI 10.5281/zenodo.22552056
(Apache-2.0 / CC BY 4.0). Full text also in paper/.
Citation
@misc{chauhan2026bala,
title = {Distilling Open Reasoning Corpora into a 3B-Active MoE on Commodity GPUs: What Transfers, What Doesn't, and a Decontamination Check},
author = {Chauhan, Tushar},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.22552056},
url = {https://doi.org/10.5281/zenodo.22552056}
}