thox-micro-125m-instruct
Instruction-tuned ThoxMicro-125M: the THOX-owned 123.6M decoder-only base, given a ChatML turn structure and a real stop token so it answers and stops instead of continuing text. Trained 2026-09-25 on one RTX 4060 Ti 16 GB.
GGUF builds (Q4_0 device target, Q4_K_M, F16):
Thox-ai/thox-micro-125m-instruct-GGUF.
What it is, and what it is not
It is a device-role responder. With its system prompt it states what it is, states correctly that the user's data stays on the device, and stops. That is the job the THOX device catalog assigns to the 125M tier; anything conversational routes over the mesh to a larger model.
It is not a general assistant. Without its system prompt it is a weak
generalist: it produces list shape without list content, answers "what
colour is the sky" with prose, and gets 17 + 25 wrong. That is the capacity of
a 125M base with a 2.61 pretraining validation loss, and more SFT on this base
will not change it.
Prompt format
ChatML. <|im_start|> = 50257, <|im_end|> = 50258 (EOS and pad). Both reuse
unused rows of the 50,304-token vocabulary, so the model and its quantisations
are the same size as the base.
<|im_start|>system
You are a THOX model running locally on THOX hardware. Your AI. Your Data. Your Rules.<|im_end|>
<|im_start|>user
Who are you?<|im_end|>
<|im_start|>assistant
The chat template is embedded in tokenizer_config.json and in every GGUF.
Training
| Base | Thox-ai/thox-micro-125m (LlamaForCausalLM, 123,587,328 params) |
| Method | full fine-tune, fp32 master weights, bf16 autocast |
| Loss | assistant spans only, including the closing `< |
| Data | 29,868 conversations: HuggingFaceTB/smoltalk (Apache-2.0) + 171 THOX identity / data-locality / scope rows (0.57%) |
| Steps | 1,000 at effective batch 32, max_len 768 |
| Optimiser | AdamW (0.9, 0.95), wd 0.1, lr 2e-4 peak, cosine, 3% warmup |
| Hardware | 1x RTX 4060 Ti 16 GB, 32.8 min wall clock |
| Best val loss | 1.9014 (ppl 6.70) on 301 held-out conversations |
smoltalk is the corpus SmolLM2-360M-Instruct was tuned on, so the comparison below measures the base model rather than the data.
Measured head-to-head
One llama.cpp CPU build, 4 threads, temp 0, same seven prompts, same system prompt. Measured on an x86_64 dev box, not a device. Only the ordering between columns transfers.
| untuned base Q4_0 | this model Q4_0 | SmolLM2-360M-Instruct Q4_0 | |
|---|---|---|---|
| File | 81,319,168 B | 81,319,136 B | 229,118,976 B |
| Peak RSS | 190 MB | 191 MB | 473 MB |
| Decode tok/s | 460 | ~450 | ~163 |
| Stops on EOS (with system prompt) | 0 / 7 | 7 / 7 | 1 / 7 |
| Identity | "a senior at the University of Illinois" | correct | invents "Echo" / "EVE" |
| "Is my data leaving this device?" | unrelated jQuery | "No. I run locally; your data does not leave the device." | wrong without a system prompt |
| General instructions (list, one word) | no | no | yes |
Known issue: system-prompt conditioning
With the exact training system prompt above, the model answers every prompt, including general ones, with its identity line. That prompt appears on every THOX training row and on no smoltalk row, so it learned the prompt as a switch. Use a different system prompt, or none, for general requests; the data-locality answer stays correct either way. The next data mix attaches the THOX system prompt to ~10% of general rows to remove the switch.
Intended use
On-device THOX identity, status and data-locality responses on ThoxMini-class hardware, with general requests routed to a larger model. Not yet validated on a Pi Zero 2 W; the RSS and throughput above are x86 figures.
Out of scope
General chat, factual QA, arithmetic, code, and any safety-critical use.
License
Apache-2.0, as for the base model. smoltalk is Apache-2.0. Copyright (c) 2026 THOX.ai LLC. Base model inventors: Craig Ross and Phamy Xaypanya (THOX IP-017).