amx-moe-eda-w2048-bugfixinit
A 30M-parameter block-routed MoE (3.3M active per token) fine-tuned to write C functions for a small EDA-tooling corpus. Trained entirely on one Intel AMX CPU core.
This release differs from amx-moe-eda-p250x4 in two ways, both measured rather than assumed:
- Attention window 2048 instead of 256.
- Initialised from a bug-fix curriculum on real C rather than from the base pretrain.
What it can and cannot do
Read this before using it. On 198 held-out EDA blocks -- programs it never trained on -- it reaches 1.2269 nats and 0% exact match. Of 61 held-out blocks compiled after substitution into the real program, 20.5% compile and 12.8% both compile and reference anything outside their own signature. It does not write correct C from a specification. It is a research artifact for studying what a CPU-trained model of this size does, not a coding assistant.
A note on the compile numbers
Two conventions appear below, and each table says which it uses.
The nrl_main block has a median target of 1,107 tokens against a 320-token
generation cap, so every one of its generations is truncated by the harness
before it can finish. Those score 0% by construction rather than by failure,
and they are 36% of the held-out set.
- Excluding
nrl_main: 20.5% compile, 12.8% work. What the model does on work it can actually complete. Used in the summary above. - Including it: 13.1% compile, 8.2% work. Used in the comparison with the previous release, because that release's numbers were measured the same way and changing one side only would be misleading.
Any comparison with other numbers from this project should check which convention they used.
Measured results
Held-out EDA blocks, 198 targets, each arm at its own best checkpoint:
| init | window | held-out CE |
|---|---|---|
| 4-day base | 256 | 1.3358 |
| 4-day base | 2048 | 1.3054 |
| bug-fix curriculum | 2048 | 1.2269 |
Paired against the 4-day init at window 2048, on identical blocks:
-0.0443 nats, t = -3.02, significant; wins on 122/198 blocks (62%)
Compile rate on 61 held-out blocks, against the 4-day init:
| 4-day init | this model | paired p | |
|---|---|---|---|
| compiles | 8.2% | 13.1% | 0.508 |
| compiles and is not a stub or hollow | 6.6% | 8.2% | 1.000 |
| hollow | 6.6% | 14.8% | 0.125 |
The compile advantage is not established, and its extra compiles are disproportionately hollow -- bodies that reference nothing outside their own signature. The cross-entropy gain is the real one.
Comparison with the previous release
The prior model is gdiamos/amx-moe-eda-p250x4. Both are the same architecture and the same 250-program corpus; this one adds window 2048 and the bug-fix init.
The two releases were selected by different criteria, and that matters more than the architecture difference. p250x4 published its step-20,631 checkpoint deliberately -- its card calls it "the fitted end of that trade", chosen because fit buys compile rate even as held-out loss gets worse. This release publishes step 4,125, chosen by held-out cross-entropy. So the honest comparison is two tables, not one.
As published: what you get if you download each
Both rows INCLUDE nrl_main, for parity with the previous release.
All numbers re-measured here on the same 61-block held-out set, so they are comparable to each other (they differ slightly from the p250x4 card, which used n=67 and its own eval run).
| p250x4 (step 20,631) | this model (step 4,125) | |
|---|---|---|
| held-out cross-entropy | 1.6892 | 1.2269 |
| blocks that compile | 8.2% | 13.1% |
| compile and not stub/hollow | 6.6% | 8.2% |
| hollow | 1.6% | 14.8% |
| exact match | 0% | 0% |
Better on loss and on raw compile rate, and noticeably worse on hollow output -- nearly one in seven of this model's held-out generations compiles while referencing nothing outside its own signature, against one in sixty for p250x4. If you are counting compiles as successes without the guards, this model will flatter itself more than the old one did.
Controlled: both at step 4,125, same eval
Both rows INCLUDE nrl_main.
Strip out the checkpoint-selection difference and the picture narrows:
| p250x4 arm | this model | |
|---|---|---|
| held-out cross-entropy | 1.3358 | 1.2269 |
| blocks that compile | 14.8% | 13.1% |
| compile and not stub/hollow | 9.8% | 8.2% |
Matched checkpoint for checkpoint, the gain is cross-entropy only. On compile rate the older configuration is nominally ahead, and a paired test over the same blocks puts the difference at p=1.000 -- i.e. no difference either way. Neither window 2048 nor the bug-fix init is shown to produce more compilable C on this task.
What is actually established
- Cross-entropy: −0.0443 nats against the same-architecture 4-day init, t = −3.02, winning on 122/198 blocks. Significant and broad.
- Compile rate: nothing. Six paired tests across these arms, none reaching p < 0.05 at n≈61 with 5–13 events per arm.
- The reason the window helps less than its mechanism suggests: only 34.5% of EDA target lines appear verbatim in the prompt, against 96% on the bug-fix task. Retrieval pays only where there is something to retrieve.
Why the window is 2048
The copy circuit in this model family is hard-capped at the training window. Asked to repeat a literal span, the window-256 model scores 74.9% token accuracy at distance 259 and 0.2% at 367 -- a step function, not a decay. Held-out EDA block prompts have a median length of 882 tokens, so at window 256 roughly 72% of the skeleton a block must copy identifiers from was outside the model's reach.
Widening it helps less here than the mechanism suggests, because only 34.5% of EDA target lines appear verbatim in the prompt (against 96% on the bug-fix task). Retrieval only pays where there is something to retrieve.
Why a bug-fix init
The bug-fix curriculum -- 24,539 distinct C functions with one injected bug each -- was built to install a copy circuit and failed at that goal: the copy probe was unchanged at every checkpoint. It still transfers usefully here, most likely as 15k functions of real C rather than as a circuit.
Training
| parameters | 30,029,429 total / 3,315,744 active per token |
| architecture | 6 layers, [lin, swa, swa, lin, swa, lin], d_model 256 |
| MoE | 64 experts, k=4, 2 shared, route_block 256 |
| window | 2048, chunked |
| hardware | one Intel AMX CPU core |
| init | bug-fix window-2048 model, itself from a 1.75B-token pretrain |
| fine-tune | 84.5M tokens; best checkpoint is step 4,125, not the final |
The released checkpoint is step 4,125. The final step 20,631 checkpoint scores 1.5251 -- 24% worse -- because 200 passes over 250 programs is far past the optimum. Anyone reproducing this should early-stop.
Known limitations
- 0% exact match on held-out blocks. Generation derails over long outputs: per-token accuracy on copyable spans is about 52%, which compounds.
- Hollow outputs. 14.8% of held-out generations compile while computing
nothing. Check outputs against the
stub/hollowguards before counting a compile as a success. - Trained on one narrow corpus. It speaks this EDA dialect and its
nrlruntime, not C in general. - Greedy decoding derails; sampling and repetition penalties were swept and none produced a significant improvement.
License
Apache 2.0.