RetinalAI compact narrator (135M, vocabulary-pruned)
Writes the free-text clinical narrative for a retinal screening result. It is a research artifact, not a medical device — see Limitations.
Distilled (sequence-level KD) from a self-hosted Qwen/Qwen3-8B-AWQ teacher onto
SmolLM2-135M-Instruct, then vocabulary-pruned for footprint.
What is different about it
- Prose only. It emits the narrative as plain text. The structured triage decision is served separately by a 3 KB linear head, so this model has no JSON contract to break. An earlier JSON-emitting version of the same base model failed to parse on 100% of cases once 4-bit quantized; prose-only generation is 1.000 at every precision.
- Pruned vocabulary, 49,152 -> 920 tokens. Embedding rows are the original
tokenizer ids listed in
keep_ids.json, in order. The tokenizer is unmodified; callers map ids old->new before the model and new->old before decoding. Tokens outside the kept set are re-encoded byte-wise rather than dropped. The vocabulary is seeded from all 45 disease names the upstream classifier can emit, and all 45 are verified to survive the round trip.
Use src/narrator/compact.py from the OptiscanAI repository — the id remap is
required, so AutoModelForCausalLM alone will produce garbage.
Evaluation
24 held-out cases, scored against the teacher's own narrative for the same case:
| precision | size | generation rate | omission | misquoted probabilities | acuity divergence |
|---|---|---|---|---|---|
| bf16 | 213.5 MB | 1.000 | 0.000 | 0.000 | 0.250 |
| int8 | 107.3 MB | — | — | — | — (37.5 s/case, impractical) |
| nf4 | 54.2 MB | 1.000 | 0.417 | 0.167 | 0.583 |
bf16 is the only precision recommended. 4-bit reaches 54 MB but drops findings the teacher reported in 42% of cases while remaining fluent, which makes the omissions hard to notice.
"Acuity divergence" counts narratives using urgency language the teacher did not use for that same case. It is a broad screening signal, not a hallucination count.
Limitations
- Not a medical device. Not for clinical use. No regulatory clearance, no clinical validation, no prospective study.
- Evaluated on 24 held-out cases from a single RFMiD-derived sample. That is enough to distinguish "works" from "broken" and little else.
- No clinician review of generated text has been performed.
- Trained on 56 examples. It reproduces the teacher's register on cases that look like its training distribution and has not been probed outside it.
- No EMERGENCY case appeared anywhere in the source data, so emergency phrasing is untested. Escalation is handled deterministically upstream, not here.
- Callers must append their own AI-disclosure statement; the teacher traces did not contain one, so the model does not produce it.