lid-neural-25.2
lid-neural-25 is a language identification model covering 25 languages,
fine-tuned from xlm-roberta-base as a sequence classifier. Trades fastText's
CPU-only footprint for meaningfully higher accuracy, especially on short,
harder-to-classify text.
- 98.2% accuracy on short text (questions/queries) — vs. 97.3% for the fastText sibling, with the gap concentrated in exactly the languages that matter most (see Benchmarks).
- Needs a GPU to fine-tune, CPU-fine for inference.
- Two checkpoints for two input lengths, same split as the fastText sibling.
For a much smaller, CPU-only alternative at a small accuracy cost, see
lid-lite-25 (fastText).
🗒️ Model Details
| Model | Trained on | Use for |
|---|---|---|
lid-neural-25.1 | Long-form text (paragraph-length) | Documents, articles, passages |
lid-neural-25.2 | Short text (sentence/question-length) | Search queries, chat messages, short user input |
| Base model | xlm-roberta-base |
| Parameters | 125M |
| Max sequence length | 128 tokens |
| Languages | 25 (see below) |
| Training data | olaverse/qg-passages-multi |
| Training | 2 epochs, batch size 16 × grad-accum 4, lr 2e-5 |
Languages: af am de en fr ha hi ig id it ja ko nl pl pt ru sn so es sw tr vi xh yo zu (ISO 639-1; ISO 639-3: afr amh deu eng fra hau hin ibo ind ita jpn kor nld pol por rus sna som spa swh tur vie xho yor zul)
🏃 Usage
from transformers import pipeline
clf = pipeline("text-classification", model="olaverse/lid-neural-25.2") # or .1 for passages
result = clf("What causes ocean tides?")
print(result) # [{'label': 'eng', 'score': 0.999...}]
Use .1 for long-form text, .2 for short queries — see Model Details above.
📊 Benchmarks
Held-out validation split (5% of training data, not seen during training),
same split used to evaluate lid-lite-25 for direct comparison.
lid-neural-25.1 (passages) — overall accuracy: 99.9% (n=2,498)
Matches the fastText sibling at this input length. All languages ≥ 0.989 F1.
lid-neural-25.2 (questions) — overall accuracy: 98.2% (n=7,419)
| Language | lid-lite-25 F1 | lid-neural-25.2 F1 | Δ |
|---|---|---|---|
| amh | 0.972 | 0.992 | +0.020 |
| jpn | 0.911 | 0.997 | +0.086 |
| xho | 0.780 | 0.786 | +0.006 |
| zul | 0.769 | 0.789 | +0.020 |
| (remaining 21 languages) | ≥0.983 | ≥0.993 | small, consistent |
The neural variant closes most of the fastText model's short-text gaps — notably Japanese (+8.6 F1 points) and Amharic — but does not fully resolve the Zulu/Xhosa confusion, which persists at a similar magnitude in both architectures.
⚠️ Known limitations
- Zulu/Xhosa confusion on short text persists even with a much larger,
fine-tuned transformer — both score F1 ~0.79 on
lid-neural-25.2, versus ≥0.98 for every other language. Since two structurally different model families (linear n-gram classifier vs. transformer) show the same failure pattern, this looks like a genuine linguistic difficulty (Zulu and Xhosa are closely related Nguni languages with substantial shared vocabulary and orthography) rather than a fixable model weakness — treat predictions between these two specifically with reduced confidence on short input. - Same training-data caveats as
lid-lite-25: machine/teacher-generated text, untested on human social media or code-switched input. - Requires a GPU to fine-tune further; inference is CPU-feasible but slower
than
lid-lite-25.
Training data & licensing
Fine-tuned from xlm-roberta-base (MIT) on
olaverse/qg-passages-multi
(Apache-2.0). Released under Apache-2.0.
Citation
@misc{lid-neural-25,
title = {lid-neural-25},
author = {Olaverse},
year = {2026},
url = {https://huggingface.co/olaverse/lid-neural-25.1}
}