team03 Math SLM — v4 (GATE + General Mathematics)
A domain-specific small language model for step-by-step math problem solving, built by team03 (SLM Learners) for the Pramana SLM++ Bootcamp Round 2 submission.
- Base: Qwen2.5-0.5B-Instruct (494M params, fine-tuned — not from scratch)
- Method: QLoRA (r=16, 4-bit NF4 base, bf16 compute), merged into the base weights
- Domain: GATE-style + general mathematics
- Intended users: GATE / competitive-exam aspirants and math students
- Use case: free-form math QA — direct answers for simple prompts, chain-of-thought + "Final answer:" for complex ones
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"singhmandavi/math-slm-qwen2.5-0.5b-v4", torch_dtype=torch.float16)
tok = AutoTokenizer.from_pretrained("singhmandavi/math-slm-qwen2.5-0.5b-v4")
msgs = [{"role": "user", "content": "What is 17 * 23? Solve step by step."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=1024, temperature=0,
pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
For an OpenAI-compatible endpoint, serve with serve_hf.py (stdlib + transformers only, no Ollama needed).
Training details
| Item | Value |
|---|---|
| Corpus | train_all.jsonl — 29,853 rows (14,308 direct + 15,545 CoT) |
| Effective train / val | 23,532 / 471 (after --drop-overlong, 5,850 CoT rows ≥2048 tokens dropped) |
| Sources | ExamBench (169Pi/exambench, Apache-2.0, commit b5f3d8b) + MathNet IMO split (CC-BY-4.0) |
| Optimizer | AdamW (betas 0.9/0.999, eps 1e-8, wd 0.01) |
| LR | 1.5e-4, cosine decay, ~132 warmup steps (3% of total) |
| Effective batch | 16 (2 micro-batch × 8 grad-accum) |
| Epochs / steps | 3 / 4,410 logged (theoretical total 4,413 = 1,471/epoch × 3; last multiple of 5 logged) |
| Max seq len | 2,048 |
| LoRA | r=16, alpha=32, dropout 0.05, all proj targets (q/k/v/o/gate/up/down); 8.8M trainable params (1.75%) |
| Hardware | 1× NVIDIA H100 80GB (Lightning Studio), ~1.84 GPU-hours |
| Seed | 7 |
| Tokens seen | 53,356,887 (last logged, step 4,410) |
Precision note: training ran in bf16 compute (QLoRA 4-bit NF4 base), but the merged checkpoint uploaded here is float16 (the merge step reloads the base in fp16).
Data provenance & decontamination
- Public Hugging Face datasets pulled via
pull_data.py; licenses verified through the HF API on 2026-09-05 and recorded indata_manifest.md. - Decontamination: 13-gram word overlap (GPT-3/Chinchilla style) + MinHash-LSH Jaccard (threshold 0.5) against held-out eval sets — 0 exact matches / 0 8-gram hits vs the held-out set, and 0 overlap vs IMO 2025.
- Held-out eval set built with
build_heldout_eval.pyfrom raw ExamBench rows never used in training, with a final overlap check that fails loudly on any contamination. - Caveat (documented, not hidden): the GATE-PYQ-specific contamination check was blocked (IIT-KGP site down at the time) and never re-run; and a byte-for-byte SHA-256 cross-check between the Kaggle-side and Studio-side corpus copies was not performed — only the row count (29,853) is confirmed to match.
Evaluation results
Level 1 — training health (results_level1_v4.json)
| Check | Result |
|---|---|
| Numerical stability | PASS |
| Undertraining | PASS (loss reduction 19.7%) |
| Overfitting | OK (final gap 0.14, stable) |
| Gradient explosion | OK |
| Convergence | FAIL (marginal — loss trend ticks up in the final 10% of steps; consistent with cosine-decay/batch noise rather than real instability, since all other checks are healthy) |
Level 2 — general capability / instruction following (results_level2_v4.json)
- Instruction-following score I = 8/33 (24.24%).
- Main failure mode: the model prepends a "here is the thinking process..." CoT preamble even under strict format constraints (one-word, JSON-only, exact word counts), despite usually getting the underlying answer right. This is inherited from the CoT-heavy training mix and is the main quality gap.
Level 3 — not run standalone for this model.
Limitations
- Weak instruction-following under strict output-format constraints (see Level 2 above).
- Convergence check fails marginally (noisy end-of-run loss slope); training itself was otherwise healthy.
- Verbose-CoT bias — mitigated but not eliminated by including 48% direct-format rows.
- Trained at 2,048 tokens context; longer problems are truncated rather than handled.
Reproducibility
- Training log:
runs/math_slm_v4/training_log.jsonl(official L1 schema: step, train_loss, val_loss, lr, grad_norm, tokens_seen) - Training events:
runs/math_slm_v4/training_events.jsonl - Checkpoints:
best(by val loss) + merged final; run completed cleanly (checkpoint timestamps match the log's final timestamp, 2026-09-17 18:28) - Exact dependency list:
dependency_versions.txt(211 packages,pip freezefrom the GPU node) - Corpus hash (SHA-256):
eb4fecba0468286afe3e4960c8680c45f87e46d414941d0614dfe2f5ad518be5
License
Base model Qwen2.5-0.5B-Instruct is Apache-2.0. Training data: ExamBench (Apache-2.0) and MathNet (CC-BY-4.0, attribution required).