tinyLM-8M-exp
Tiny 8M-class Qwen3-config causal LM with math-only novelty-gated GQA.
Architecture
| Item | Value |
|---|
| Config type | tinyqwen3_novelty |
| Parameters | 8.132M |
| Layers | 8 |
| Hidden size | 256 |
| MLP size | 896 |
| Query heads | 8 |
| KV heads | 4 |
| Head dim | 32 |
| RoPE theta | 2500 |
| Tied embeddings | yes |
| Attention | Value |
|---|
| Type | GQA |
| Novelty gate | math-only element-wise RMS-normalized abs-delta |
| Gate floor | 0.05 |
Training
| Item | Value |
|---|
| Tokenizer | AxiomicLabs/GPT-S2-5M |
| Sequence length | 512 |
| Microbatch size | 1024 |
| Gradient accumulation | 4 |
| Effective batch size | 4096 |
| Steps | 10,000 |
| Validation cadence | every 1,000 steps |
| Official lm-eval | after final Hub upload on ARC-Easy, ARC-Challenge, PIQA, HellaSwag |
| LR schedule | warmup, cosine to min by 10,000 |
| Optimizer | Muon for middle 2D weights, AdamW for the rest |
| Special-token policy | BOS/EOS are document-level; `< |
| Dataset | Share | Config |
|---|
HuggingFaceFW/fineweb-edu | 60.0% | sample-100BT |
HuggingFaceTB/smollm-corpus | 30.0% | cosmopedia-v2 only |
epfml/FineWeb-HQ | 10.0% | default |
Validation
| Metric | Value |
|---|
| Dataset | Salesforce/wikitext, wikitext-103-raw-v1, validation |
| Context / stride | 512 / 256 |
| Loss | 3.2769 |
| Perplexity | 26.49 |
| UTF-8 BPB | 1.4992 |
| Scored tokens | 365,258 |
| UTF-8 bytes | 1,151,766 |
Evaluation
Scores were run after Hub upload against revision
d95a00a6edafab4bc2d6b60a28e6893b00f52699.
ARC-Easy, ARC-Challenge, PIQA, and HellaSwag use official lm_eval
0-shot log-likelihood scoring. ArithMark-2.0 uses the same continuation
NLL scoring style with a custom scorer because it is not available in
lm_eval.
| Task | n | acc | acc stderr | acc_norm | acc_norm stderr |
|---|
| ARC-Easy | 2,376 | 37.04% | 0.99% | 35.86% | 0.98% |
| ARC-Challenge | 1,172 | 18.77% | 1.14% | 22.87% | 1.23% |
| PIQA | 1,838 | 57.67% | 1.15% | 57.89% | 1.15% |
| HellaSwag | 10,042 | 26.88% | 0.44% | 27.88% | 0.45% |
| ArithMark-2.0 | 2,500 | 25.12% | 0.87% | 24.44% | 0.86% |
Load And Generate
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "User01110/tinyLM-8M-exp"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo,
trust_remote_code=True,
torch_dtype="auto",
device_map="auto",
)
prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
print(inputs.input_ids[0][:2].tolist()) # auto-prefix: [<|im_start|>, <bos>]
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.65,
top_k=30,
repetition_penalty=1.2,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
This repo uses a self-contained remote TinyQwen3NoveltyConfig plus model code for a Qwen3-style dense decoder with a math-only novelty-gated attention block.