deka-qwen3-emb-0.6b-multipos-fact
Full fine-tune of Qwen/Qwen3-Embedding-0.6B for retrieving prior Thai Supreme Court decisions (คำพิพากษาฎีกา) from a 9,998-case corpus. Anonymous release accompanying a paper under review.
Objective
Supervised contrastive learning in the L_out^sup form of
Khosla et al. (2020) — the average over positives sits
outside the log — applied to retrieval. Each anchor gets 4 document slots
[easy_pos, hard_pos, hard_neg, easy_neg], and when a distinct hard positive exists both
positive slots count:
L = -1/|P| * Σ_{p∈P} log softmax(sim(q, d_p) / τ)
The denominator pools every document in the batch (16 anchors × 4 = 64 docs), with known positives of other anchors masked out; negatives are mined hard/easy negatives plus in-batch negatives, the setup of Karpukhin et al. (2020). Distinct from SupMPN (Dehghan & Amasyali, 2022), which assembles multiple positives and negatives for sentence-level STS: here positives come from case-law relevance judgements (at most two per anchor, mined and incomplete) and the target is corpus-scale retrieval.
Data, pooling, evaluation and model selection are shared code with the InfoNCE / triplet / weighted-hard-negative checkpoints in the same sweep; the loss above is the only difference.
Config
| Fine-tuning | full, all 596M params (no adapter) |
| Pooling | last token (left padding) + L2 normalize, dim 1024 |
| Query side | instruction prefix; document side raw |
| τ / lr / batch | 0.05 / 2e-5 AdamW / 16 anchors |
| max_len | 1300 doc, 256 persona query |
| Training register | fact — queries are the fact section of a case (symmetric setting) |
| Stopping | dev R@10 over the full corpus, patience 3 → stopped at epoch 5, best epoch 2 |
Usage
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("anonymousresearch123/deka-qwen3-emb-0.6b-multipos-fact")
q = m.encode(["จำเลยขับรถโดยประมาทเป็นเหตุให้ผู้อื่นถึงแก่ความตาย"], prompt_name="query")
d = m.encode(["<ข้อเท็จจริงของคำพิพากษาฎีกา ...>"]) # no prefix on documents
m.similarity(q, d)
With plain transformers, keep padding_side="left" and pool the last position — right
padding returns the embedding of a pad token and fails silently.
Results
105 test queries, full 9,998-case corpus, clean qrels, from anonymousresearch123/deka-retrieval. One model, three query registers:
Lay (layperson phrasing), Lawyer, Fact (the case's own fact section).
R@10 / nDCG@10 / MRR×100:
| Lay | Lawyer | Fact | |
|---|---|---|---|
| BM25 | 6.19 / 4.31 / 6.60 | 22.86 / 15.23 / 18.43 | 26.19 / 16.34 / 20.48 |
| Qwen3-Embedding-0.6B, zero-shot | 7.62 / 4.58 / 7.45 | 20.00 / 11.89 / 15.46 | 26.67 / 17.46 / 24.64 |
| this model | 11.90 / 6.26 / 6.52 | 21.90 / 13.57 / 16.64 | 31.43 / 20.88 / 23.87 |
Recall improves over zero-shot (+4.76 R@10 on Fact, +4.28 on Lay) while MRR drops on both:
more relevant cases enter the top-10, the single best hit ranks lower.
training_history.json holds the per-epoch log, R@5/50/100 and the full config.
Limitations
- 105 test queries: one query is worth 0.95 R@10 points, so gaps of 1–2 points are noise.
- One seed, one run per cell. Retrieval replays bit-identically, which measures pipeline determinism, not seed variance.
- Trained and model-selected on fact-register queries; on layperson phrasing R@10 falls to 11.9.
- Positives were mined and are incomplete, so recall is a lower bound, unevenly across queries.
- Documents over 1300 tokens are truncated.
- Retrieval only. Similarity is not precedential authority, and nothing here checks whether a retrieved case is still good law.