litil-embed-0.6b
A 0.6B legal retrieval embedding model. It is Octen/Octen-Embedding-0.6B fine-tuned on legal retrieval data and distilled from Octen/Octen-Embedding-8B, then merged back toward the base. Same architecture, tokenizer, prompts and 1024-dim output as the base, so it is a drop-in replacement.
MTEB(Law, v1)
nDCG@10, mteb 2.20.5, same prompts and settings as the base.
| Task | Octen-Embedding-0.6B | litil-embed-0.6b |
|---|---|---|
| AILACasedocs | 59.95 | 56.28 |
| AILAStatutes | 89.60 | 89.53 |
| GerDaLIRSmall | 41.08 | 45.20 |
| LeCaRDv2 | 73.82 | 73.96 |
| LegalBenchConsumerContractsQA | 81.41 | 82.03 |
| LegalBenchCorporateLobbying | 94.77 | 95.64 |
| LegalQuAD | 65.82 | 70.45 |
| LegalSummarization | 72.86 | 73.34 |
| Mean | 72.41 | 73.30 |
Gains are largest on German legal retrieval. AILACasedocs is below the base; the training set has no Indian case law.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("litillabs/litil-embed-0.6b")
queries = ["Instruct: Retrieve text based on user query.\nQuery: Kündigungsfrist bei befristetem Mietvertrag"]
documents = ["§ 542 BGB Ende des Mietverhältnisses ..."]
q = model.encode(queries, normalize_embeddings=True)
d = model.encode(documents, normalize_embeddings=True)
scores = q @ d.T
Queries take an instruction prefix (Instruct: ...\nQuery: ); documents take none. Use the same instructions as the base model for a given task. Cosine similarity. Context up to 32k tokens; we evaluated at 18,480.
Training
- Base:
Octen/Octen-Embedding-0.6B(revision1a00a4e8). - Teacher:
Octen/Octen-Embedding-8B(revision5adcfa29), used to mine hard negatives and to provide relevance scores for distillation. - Objective: InfoNCE with 4 teacher-mined hard negatives per query plus in-batch negatives, and a KL term against the teacher's score distribution.
- LoRA rank 32 on attention and MLP projections, batch 64, learning rate 2e-5, 855 steps, 35% general-retrieval replay. The adapter was merged into the base at half strength.
Data
| Source | Rows | Licence |
|---|---|---|
| GerDaLIR train split | 20,000 | MIT |
| CUAD-QA, PolicyQA, PrivacyQA (question → clause) | 11,993 | CC-BY-4.0, MIT, MIT |
| BillSum (title → summary) | 8,000 | CC0 |
| LeCaRDv2 train split, LEAD | 2,554 | MIT |
| German federal statutes (heading → section) | 2,150 | ODbL |
| Natural Questions, GermanQuAD, MIRACL zh (general replay) | 10,000 | CC-BY-SA-3.0, CC-BY-4.0, Apache-2.0 |
All rows were checked against the eight MTEB(Law, v1) test sets for exact and near-duplicate overlap before training. GerDaLIR and LeCaRDv2 train splits share source corpora with GerDaLIRSmall and LeCaRDv2; the model is therefore not zero-shot on those two tasks and declares both as training datasets.
Limitations
Trained for English, German and Chinese legal retrieval. Indian case retrieval is weaker than the base. Not trained for classification, clustering or STS.
Licence
Apache-2.0, same as the base model.