MiniLM-L6-v2 LoRA — Tax & Finance Retrieval
A parameter-efficient retrieval encoder fine-tuned from
sentence-transformers/all-MiniLM-L6-v2
for Indian tax and finance question-answering retrieval.
The model maps a taxpayer's question to the exact statute passage (or financial document) it is answered by. It is compact (22M params, ~90 MB), fast to embed, and cheap to deploy — designed as a retrieval unit for a RAG pipeline on modest hardware.
Key facts
| Property | Value |
|---|---|
| Base model | all-MiniLM-L6-v2 (22.7M params) |
| Fine-tuning | LoRA (r=16, α=32, dropout 0.05) on query/key/value/dense |
| Trainable params | 675,840 — 2.89% of the model |
| Embedding dim | 384 |
| Max sequence length | 256 tokens |
| Loss | MultipleNegativesRankingLoss |
| Compute | GPU (Kaggle), P100 |
| License | Apache-2.0 (base model) |
Headline results
Measured on a held-out pooled retrieval test of 839 queries scored against the full combined tax + finance corpus (100% unseen during training). Reported as mean ± std across 3 independent training seeds (42, 123, 456).
| Metric | LoRA fine-tuned | Zero-shot baseline | Δ |
|---|---|---|---|
| nDCG@10 | 0.592 ± 0.001 | 0.447 | +0.145 |
| MRR@10 | 0.484 ± 0.002 | 0.361 | +0.123 |
| MAP@100 | 0.488 ± 0.002 | 0.373 | +0.115 |
| Recall@1 | 0.327 ± 0.002 | 0.250 | +0.076 |
| Recall@5 | 0.685 ± 0.001 | 0.496 | +0.189 |
| Recall@10 | 0.948 ± 0.002 | 0.735 | +0.212 |
| Accuracy@1 | 0.327 ± 0.002 | 0.250 | +0.076 |
Fine-tuning delivers a +32% relative improvement in nDCG@10 over the untrained base, with high seed-to-seed stability (std ≤ 0.002).
Eval protocol (research-grade)
- Stratified splits (by tax topic) — train / val / test = 6,708 / 838 / 839; no leakage.
- Elbow-based epoch selection on the validation set only; the test set is scored exactly once.
- 3 seeds, aggregate mean ± std.
- Zero-shot baseline on the identical held-out test.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("vivekkopthsd/minilm-l6-v2-taxfinance-lora")
query = model.encode(
["What is the deduction for rent paid on salary income?"],
normalize_embeddings=True,
)
docs = model.encode(
[
"Section 134: Deduction in respect of rent paid.",
"Section 123: Deduction for life insurance premium.",
],
normalize_embeddings=True,
)
scores = query @ docs.T
Training
Data — pooled from two sources:
ligaments-dev/indian-income-tax-qa— 6,800 (question, answer) pairs from Indian Income-Tax statutes, used as (query, positive-document) pairs.AIR-Bench/qa_finance_en(AIR-Bench_24.04) — 1,585 financial texts, used as self-supervised pairs.
Objective — MultipleNegativesRankingLoss: for each (query, positive) pair
in a batch, the positive is treated as the correct answer while all other
documents in the batch act as in-batch negatives. This is the standard
contrastive objective for dense retrieval.
Hyperparameters
| Parameter | Value |
|---|---|
| Optimizer | AdamW (weight decay 0.01) |
| Learning rate | 2e-4 |
| Batch size | 32 |
| Scheduler | cosine with 10% warmup |
| LoRA rank / α | 16 / 32 |
| LoRA dropout | 0.05 |
| Epochs | 15 (elbow-selected ~4–5) |
| Max grad norm | 1.0 |
| Seeds | 42, 123, 456 |
Limitations
- English only. The base model has limited multilingual capacity; retrieval quality degrades sharply for Hindi or code-mixed Hinglish input. For multilingual retrieval, a multilingual-base fine-tune is required.
- Domain-specific. Trained on Indian income-tax Q&A and generic finance; generalization to unrelated legal or technical domains is not guaranteed.
- Short context. 256-token limit suits statute sections and FAQ-style documents, not long passages.
- The finance subset uses self-supervised (text→text) pairs, so pooled metrics should be interpreted alongside the tax-only performance.
Citation
@misc{vivekkopthsd-minilm-l6-v2-taxfinance-lora,
title = {MiniLM-L6-v2 LoRA: Tax and Finance Retrieval},
author = {Vivek},
year = {2026},
url = {https://huggingface.co/vivekkopthsd/minilm-l6-v2-taxfinance-lora}
}