retrieval-a025 — Coathink writing-recall embedder
A LoRA fine-tune of Qwen/Qwen3-Embedding-0.6B, merged back into the base at
α = 0.25 (WiSE-FT style interpolation), for one task: given a sentence a
writer is drafting, retrieve the note card (saved highlight) it draws on.
model.safetensors is 1.1 GB, fp16, 1024-dim, Qwen3Model. Standard
SentenceTransformers layout — SentenceTransformer("<repo-id>") just works.
⚠️ Read this before choosing it
On our only human-judged benchmark, this model does not beat the frozen off-the-shelf models it was meant to improve on.
| model | human-gold nDCG@10 | notes |
|---|---|---|
Octen/Octen-Embedding-0.6B (native prompt) | 0.3559 | frozen, Apache-2.0 |
Qwen/Qwen3-Embedding-0.6B (no prompt) | 0.3538 | frozen |
Octen/Octen-Embedding-0.6B (no prompt) | 0.3505 | frozen |
| this model (a025) | 0.3482 | |
| v2a (pure distill, unreleased) | 0.303 |
n = 34 queries / 859 cards, one neuroscience paper labeled by its own author. Paired MDE ≈ 0.030, so the top four are a statistical tie — but a tie is the honest reading, not a win. A separate agent-labeled ruler scores this model resolvably above base (+0.026), but a cross-ruler agreement test showed that ruler disagrees with the human anchor on exactly this kind of close call, so we do not count it.
Do not use the custom Instruct: ... query prefix. A prompt ablation found
it is net-negative for the base model; frozen base with no prompt is the best
number in the table above. If you use Octen, keep its native prompt.
Where it does win: mid-sentence queries
Retrieval in the product fires when the writer pauses mid-sentence, not on a finished sentence. Evaluated in that regime (clause truncated to 40/60% with the preceding ~30 words prepended), the ranking flips and this model leads:
| operating point | base | octen | a025 |
|---|---|---|---|
| 40% of clause + context | 0.210 | 0.215 | 0.223 |
| 60% of clause + context | 0.257 | 0.272 | 0.285 |
| 100% clause, no context | 0.354 | 0.356 | 0.348 |
This is the only claim we make for this model, and it is a weak one: on a second, cross-domain ruler (a physics/CS paper, 31 queries / 1101 cards) the advantage did not replicate — a025 and base both scored 0.1795 at 40%. Treat the mid-sentence edge as unconfirmed outside the domain it was measured in.
Training
MarginMSE on citation-grounded pairs from unarXive, teacher = Qwen3-Reranker-4B
log-odds margins, LoRA r16/α32, then merged at α=0.25. The interpolation is what
made it survive out-of-domain; the pure-distill checkpoint (v2a, 0.303) is worse
than the base it started from.
Scaling this recipe fails. A 60k field-balanced set of s2orc citation pairs (12 fields, hard negatives, same teacher, same loss) produced 0.2475 from base and 0.2410 continuing from this model — a statistically significant regression (per-query AUC 0.866 vs base 0.934, gap 0.068 > MDE 0.041). The citation-proxy signal is misaligned with human writing-utility judgments; more of it does not help.
Recommendation
For a fresh integration, prefer Octen/Octen-Embedding-0.6B frozen: tied-best
or better on both rulers, Apache-2.0, 600 MB, same backbone and MLX path, and no
LoRA-merge / prompt-calibration apparatus to maintain. Reach for this model only
if you are specifically working the mid-sentence regime and want to reproduce the
table above.
No MLX build exists. a025-mlx in the source repo is a broken stub (the
safetensors entry is an 84-byte symlink). A Swift/MLX consumer needs a real
conversion first; mlx-community/Qwen3-Embedding-0.6B-4bit-DWQ is the only
ready-made MLX option today, and 4-bit was measured to cost nothing (device
goldset: 4-bit base 0.733 vs bf16 a025 0.730).