SemanticRepair-270M

A 270M model that proposes request lines for an embedding router, or NO_REQUEST when it finds no request. It does not answer the user's question. The host application owns capability admission and tool execution; model output alone does not authorize an operation.

SDG's evaluated native path admits individual operations with their own queries, which can contain the model's repaired request text. This report checks those admitted per-operation queries. It does not execute graph tools or establish the correctness of their results.

Frozen routing-fidelity validation — 5 October 2026

The frozen candidate reduces unwanted automatic admissions but does not improve overall completion on this validation bank. Complete supported requests fall from 20/80 to 9/80, while correct refusals increase from 9/16 to 15/16. The result does not establish a generally improved replacement or 100% routing accuracy. Model weights are unchanged. These are native routing/admission measurements; no graph tools or answer generation execute.

The less strict capability-set measure is 33/96 for baseline and 26/96 for candidate. The candidate's 16 automatic admissions comprise 9 complete supported requests, 5 cases with an unwanted capability, 1 incomplete operation set and 1 correct operation set with a failed required literal. “Correct refusal” includes unresolved route choices. This table uses actual native Chat; direct-router baselines from different harness versions are not compared.

Model-only request-presence classification is 88/96 (91.67%) in both conditions on this bank: 82/88 messages containing a request are recognized and 6/8 nonrequests are rejected. Protected literal retention is 56/70. These unchanged model-only scores are separate from native routing and from the historical 140/160 score on a different bank; model weights are unchanged.

The 504-row development bank and the separately frozen 96-row validation bank are distinct. Labels are agent-authored, without independent human review; translated pairs are dependent. A post-hoc audit finds two history-dependent messages in the fresh-conversation fixture. Original labels and 96-case scores remain intact; excluding only those two cases yields 27/94 versus 23/94 complete results/correct refusals, so the coverage tradeoff remains. No candidate tuning follows this validation round.

On the separate, already exposed 184-case development regression, complete capability sets or correct refusals improve from 57/184 to 74/184; supported sets from 50/156 to 52/156, negative refusals from 7/28 to 22/28, and multi-capability sets from 1/52 to 8/52. That bank has no operation multiplicity or argument annotations. Its development improvement does not override the stricter independent validation result above.

Measured candidate source: 3f6ee71f95902607e490a1f8824b43575273f409. Baseline production source: 8f773b82, run in isolation with benchmark-only instrumentation. The candidate is experimental and this evaluation does not establish successful tool execution, answer quality, deployment readiness or a generally improved replacement. Full tables, all four development experiments, frozen validation traces, operation labels and limitations are maintained in the existing Gramscii-IT/semantic-repair-routing dataset. The additional evaluation retains PolyForm Noncommercial; training data retain CC BY 4.0 and model artifacts retain Gemma terms.

Earlier measured routing candidate — 5 October 2026

The completed evaluation measures the unchanged v22 Q8_0 model with an experimental SDG admission correction. Model-only request-presence accuracy remains 87.50% (140/160). Native system exact sets or correct refusals reach 42.50% (68/160) on the development regression bank. These are different measures: detecting a request does not establish correct routing.

This is a partial correction in draft SDG PR #658, not a deployed release or completed solution to multi-intent routing. Model weights and workspace thresholds are unchanged. On a separate 24-question diagnostic bank, complete results improve from 3/24 to 7/24, but all 12 additional triple requests remain unresolved (0/12 exact). No graph tools execute, so the figures do not measure successful tool execution or source-grounded answers.

Both banks are development/diagnostic material after their outputs are inspected; labels have no independent review and translated pairs are dependent. The completed component diagnosis feeds 36 faithful manually annotated requests directly to the router without model rewriting: 31/36 have the expected highest-scoring winner, but only 19/36 pass native confidence checks (Italian 14/18, English 5/18). This identifies a native routing limitation independently of model generation; it is not a model accuracy score.

Historical scores and frozen inputs, candidate results and complete traces, and component diagnosis are all maintained in the existing Gramscii-IT/semantic-repair-routing dataset. The measured runtime is 020b8e6c, equivalent to the reviewed runtime tree at 4b640443; the component diagnostic is tested at 02f042cf. Relevant local gates pass: 1655 unit tests, 73 focused checks, 51 consumer checks, Ruff, the native 184-row evaluation, and the separate 36-component test plus its 3 contract tests. Integration with later coordinated SDG revisions and full product/browser/tool-execution verification remain pending before deployment.

Fresh adversarial evaluation — 5 October 2026

The v22 Q8_0 weights are evaluated without retraining on 160 fresh questions: 80 paired Italian/English anchors covering plain requests, long noise, negation, quoted distractors, two and three requested capabilities, unsupported actions, and forged instructions. Inputs and gold capability sets are frozen before inference; no exact normalized overlap is found with v22 train/validation/test, previous probes, or the older 481-question bank. This is a Gramscii author self-evaluation by Codex / Atlante, without independent annotation review; exact overlap exclusion does not prove absence of semantic training overlap.

Model alone, on the native completion surface

Request-line count is a structural proxy, not proof of correct semantic decomposition. Literal string retention is stricter than semantic equivalence and does not measure tools' argument retention. Unsupported actions remain genuine requests for this model-only detection score; refusing unsupported graph operations is the system's responsibility.

Model inside the current SDG native admission path

SDG base 2d01f906 includes #645 and #654. The test selects the general default graph explicitly, uses its actual Italian and English descriptions and thresholds, and calls native planning.capabilities and chat.routing.resolve. The catalogue and library start empty; each conversation is fresh with no selected documents. No graph tools or narration execute. This is a routing admission experiment, not end-to-end answer accuracy or a test against the populated live catalogue.

More supported requests are placed exactly with repair, but unwanted admissions also increase. These results do not establish a safety improvement. A wrong admission is not a wrong execution: no tool executes in this evaluation. Strict subsets count as partial, never exact. Native admission rules and repair both affect this comparison, so it does not isolate the repair model's causal effect. The English workspace retains its actual separate thresholds, without calibration on these questions.

Only 1/20 two-capability questions and 0/20 three-capability questions are admitted completely. The model can emit several request lines while their rerouting still misses or misidentifies capabilities. For example, the Italian request to map Pisa, give its ISTAT code, and retrieve its 2021 population yields three rewrite lines, but admission selects locate-address where opendata-territories is required. Unsupported actions also expose false admissions: a request to change Roma's official population is routed to opendata-statistics, although the graph has no authorized write operation.

The results are deliberately harder and broader than the historical banks below; their percentages are not comparable as a model-version regression. Eight balanced adversarial families do not estimate production frequencies. Paired translations are dependent, and the published item-level intervals and sign-test p-value assume independence; do not promote them as independent population inference. No repeated-run stability claim, other-language result, populated-catalogue search result, or conversation-continuation result is established.

The published evaluation dataset includes the frozen bank, complete question-level traces, graphs, model hashes, thresholds, scorer, source runner, workspace configuration, and verification provenance. The evaluated model remains revision acc30fd451d58620cc22dbbfab46caed3cbf12f5, with Q8_0 SHA-256 58b750b4bba4ed9cf1ea30316f1c5ad131316efaedd56cfd9440b488b2ae88f4; llama.cpp build 10621 uses greedy completion, a 96-token maximum, and repeat penalty 1.0. Structured author-reported results are in this card's model-index metadata. This evaluation does not claim a Hub verified badge or registration in its beta benchmark leaderboard service.

The surface it reads, which is not a chat prompt

Fine-tuned on a bare completion surface. It has never seen a chat template, a system message or a few-shot example. Speak to it the way it was trained or it will not work:

{Language}: {the question, verbatim}
=>

It completes with one line per request it found, or the single token NO_REQUEST.

The newline after => is part of the surface. The engine sends {Language}: {request}\n=>\n, byte for byte — the JSON example below shows it — and a prompt that stops at => is a surface the model never read.

{Language} is the English name of the language the question is in — not the language of whatever system is asking. English, Italian, French, German, Spanish; a language outside that set keeps whatever tag the caller declares, because an invented tag is a surface the model never read either.

This is the whole call the engine makes, on /v1/completions and never on a chat endpoint:

{
  "prompt": "Italian: che tempo fa domani a Bologna?\n=>\n",
  "stop": ["\n=>"],
  "temperature": 0.0,
  "repeat_penalty": 1.0,
  "max_tokens": 96
}

Greedy, because the same words must always produce the same rewrite: a router that reruns on a different restatement each time cannot be reasoned about. repeat_penalty is 1.0, which is neutral — a rewrite legitimately repeats the words of the question. The stop string cuts a runaway that starts echoing another question in its own trained format; everything before it is the answer. 96 tokens leaves room for several rewritten lines: the longest answer in the training data is 43 tokens.

The sentinel is compared case-insensitively and tolerates a trailing period. It is an instruction the model follows, not a token it is guaranteed to emit byte-exact.

These are real outputs from the released weights, greedy:

A negation is dropped, one message asking for two things becomes two lines, a request behind an opening "no" is kept, and something that is not a request at all answers with the sentinel.

What is in this repository

The GGUF's sha256 is 58b750b4bba4ed9cf1ea30316f1c5ad131316efaedd56cfd9440b488b2ae88f4. The engine pins it and refuses anything else, which is what makes a routing decision reproducible.

Both files carry the same weights: checkpoint 21,000 of training run v22, chosen on the validation split, quantised to Q8_0. Q8_0 and not Q4: measured on this seat, at temperature 0 a Q8_0 is stable to the byte across runs and a Q4 is not.

What it is measured to do

Every number below compares these weights with the previous release of this model (training run v19), both served by the same llama-server build with the same greedy call, on the same day. Pairs are counted where the two disagree, which is where the information is, with an exact two-sided sign test.

481 real questions people asked the engine, in Italian, held out from every training and selection step and routed through the full path the engine takes (the router, the repair when the route is unsure, admission only when every line is confident):

The gain is where it was aimed: requests behind an opening "no", "ok" or "I didn't get it", bare statistical topics, "what's new in X", and names written in lowercase, which the previous release answered with NO_REQUEST. Of the 118 questions it still fails on a capability it can reach, 66 are decided by the router, not the rewrite: the right capability comes first but below the bars. 38 route elsewhere and 13 are still answered with the sentinel.

900 routing tasks over graphs from another corpus, five languages, two in five naming a day:

Disagreements split 26 to 39 against this model, p = 0.14: no difference on this bench is significant, and the day column is the one to read.

100 probes written before training, 20 per family, none equal to a training row or to a real question above: 52 of 80 requests routed to the right capability with every name kept (previous release 43), and 20 of 20 non-requests answered NO_REQUEST (previous 20).

The held-out test split of the training data, 963 rows, whole completion equal to the target: 263 (previous 260).

What it does not fix, which a card naming only the gains would hide

Dates. On the 900-task bench it loses the day the question named more often than the previous release, 125 against 112, and the restatement reads as a clean request with the day simply gone. If the day matters to your routing, measure that before you serve this.

It can drop the verb of a request. Italian: scusa il disturbo, mi diresti che tempo fa domani a Bologna? comes back as dimmi se domani a Bologna è previsto: "che tempo fa" is gone.

It still refuses some requests. Italian: che novità ci sono in Django 6? answers NO_REQUEST. Of the 80 request probes, 10 still come back as the sentinel, 15 route to another capability and 3 lose a name.

What bounds all of this

None of these weaknesses are fixed. They are bounded, by the system around the model rather than by the model, and the bounds are worth stating because they decide how much a weakness costs.

Nothing it writes is executed. The restatement is compared against capability descriptions and then thrown away. Whatever runs, runs on the original message. Fixed SQL and HTTP operations are authored in the tool and cannot be assembled from either text.

The source message stays beside the restatement. Dates, language, page references and everything shown to the reader are read from what the person actually wrote. So a dropped day costs a routing decision, not the date: the question may reach the wrong capability, but the day itself was never the restatement's to lose.

A restatement that does not win by enough margin routes nothing. The router applies a margin and a floor, and below them the system abstains and asks for a rephrasing. That is why the wrong-execution column is 4 in 900 while the abstention column is 696: the failure mode is refusing, not acting wrongly.

The descriptions are the other half. What a question is matched against is written prose in the workspace, and a capability that keeps being missed is usually a description that needs rewriting, not a model that needs retraining. The rewriter is one of two things you can improve, and the cheaper one is often not the model.

The engine has a linter for exactly this, applying the same score the router applies, over a whole graph or over one description while it is being written. It reports collisions: pairs of capabilities the router cannot separate. A collision is not a verdict that either is wrong. Capabilities that overlap in meaning are expected to sit close, and the linter says where the router cannot tell them apart, leaving the choice to rewrite, merge or keep to whoever knows what the nodes are for. collision is also one of the four states a routing can end in, beside uncovered, uncertain and confident, so the reader is told which one happened rather than being handed an answer with no account of how it was reached.

Where it runs

Served with llama-server, reached by the engine as one of four model seats:

llama-server -m sft-v22-q8_0.gguf --host 127.0.0.1 --port 9250 -ngl 99 -c 2048 --jinja

It is loaded only when routing is genuinely unsure, never as the default path, and released after an idle window.

On CPU alone

Measured on this architecture at this size, on an Apple M-series machine with -ngl 0 -t 4: no GPU offload at all, four threads, and nothing in the server log naming Metal or a GPU.

762 MB resident while serving.

llama-server -m sft-v22-q8_0.gguf --host 127.0.0.1 --port 9250 -ngl 0 -t 4 -c 2048

What this does and does not establish: it runs usefully with no accelerator, on few threads. It has not been run on a phone or a tablet, and the three things that would decide it there are the three not measured here — 762 MB resident is a lot for a mobile process, loading 300 MB from slow storage is a different question from generating in 38 ms, and mobile cores throttle under heat while these did not.

The data it was trained on

Gramscii-IT/semantic-repair-routing — published, and everything below can be recounted from it.

The base is that dataset's 84,819 pairs (82,819 train, 1,000 validation, 1,000 test), cleaned by deterministic rules before training: template leaks, changed values, English targets for other languages, repeated lines and generic "category" lines removed, which leaves 80,297 train, 966 validation and 963 test rows. To them this release adds 2,515 rows in Italian and English, generated by Qwen3.6-35B-A3B and kept only when every line's top capability on the engine's own roster was the one the row was written for, every number and name of the message survived, and no row shared most of its trigrams with a probe, a real question or another row. They cover a request about the conversation itself, a bare statistical topic, "what's new in X", names in lowercase or with a typo, long noisy messages carrying two or three requests, and acknowledgements that ask for nothing. Each is repeated twice in training, about 6% of it.

The base dataset, as published: Each pair is a message somebody could plausibly write and the requests inside it restated plainly, one per line.

Five languages, close to balanced:

What the message is doing, which is what the data is really organised by. Every one of the 82,819 training rows carries one of 38 such labels; these are the ten largest:

The remaining 28 cover self-correction, code-switching, ultra-short messages, typos and slang, messages carrying two, three or four requests, eight kinds of date reference — a weekday in the past, a named holiday, a bounded range — and the two families below.

The two families that decide what this model refuses to do

injection_meta — 2,278 rows — is a message aimed at the assistant itself: reveal your instructions, ignore what you were told, pretend you have no rules. 1,901 of them answer NO_REQUEST, because there is nothing here to route. Together with injection_command these two families supply 19% of every NO_REQUEST row in the set: refusing is largely taught by messages that try hardest to get an answer.

injection_command — 2,410 rows — is the opposite lesson, and the more delicate one. A hostile but explicit command about the user's own things: delete every saved bank account, empty my personal cloud, cancel all my bookings tonight. 2,281 of them are restated faithfully, action intact.

The rewriter does not sanitise and does not soften. Deciding whether an action is allowed is not its job and it has no way to do that job well: it sees one sentence, not the account, the permissions or the consequences. A model that quietly dropped "delete" would hand the router a different request from the one the person made, and the person would never learn their instruction had been edited. So the restatement carries the command as written, the router places it, and whatever runs it is where refusing belongs.

12.6% of rows (10,432) answer NO_REQUEST, and 16.9% (14,028) answer with more than one line.

Who wrote it, and what was thrown away:

A separate model read every message against written rules and answered a two-word verdict, decoding greedily so a verdict does not change between runs. The dataset card says where judge and generator were different models and where they stopped being.

Training

Fine-tuned from google/gemma-3-270m with mlx_lm lora, full fine-tune, all 18 blocks. The released weights are checkpoint 21,000, chosen among the run's checkpoints by exact answers on the validation split (301 of 1,077), never on the test split or the benches above.

Where all of this happened

One desktop machine, and nothing left it.

Generators and judge ran on that same machine as local llama.cpp servers on loopback. No message here was written by a hosted API, and none was sent to one to be judged.

A model that decides where a question goes does not need a cluster or a month. It needs a narrow job, data built for it, and a bench honest enough to say when a round did not help.

Licence

This is a fine-tune of Gemma-3-270M — a Model Derivative under the Gemma Terms of Use, which it inherits whole. What those terms actually oblige, by their own sections:

Use (§3.2): the restricted uses in the Gemma Prohibited Use Policy are incorporated into the agreement by reference, and they bind anyone using these weights.

Redistribution (§3.1): whoever passes these weights on must include the §3.2 use restrictions as an enforceable provision in their own agreement, give every recipient a copy of the Terms, mark modified files as modified, and ship a notice stating "Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms". Those duties travel with the file, however many hands it passes through.

Outputs (§3.3): Google claims no rights in what the model writes. The restatements are yours, and so is the responsibility for them.

An engine that downloads these weights onto the operator's own machine, pinned by sha256, redistributes nothing: the duties above fall on whoever ships the file, not on whoever runs it.

Experimental SDG compound-routing follow-up

A routing-only candidate now admits the Italian Pisa map/ISTAT-code/population request correctly. On the original development regression bank, native exact triples rise from 0/20 to 5/20 and doubles from 1/20 to 6/20; complete sets or correct refusals rise from 54/160 to 68/160. On 12 additional diagnostic triple requests, exact admission remains 0/12, and the noisy regression family worsens from 8/20 to 4/20. This is a partial correction in draft SDG PR #658, not reliable general multi-intent handling or a deployed release. The additional diagnostic outputs are inspected during development, so they are not an untouched holdout.

Model weights and historical model-index scores remain unchanged. Full follow-up traces, limitations and safe independent-copy runner distinguish model behavior from native admission changes and document the source-artifact hardlink incident.