NE-LID: Northeast Language Identification

NE-LID is a sentence-level language identification model for low-resource languages of Northeast India, trained using a character n-gram fastText classifier.

The model achieves near-ceiling accuracy (99.1%) and is designed to be fast, robust, and reproducible, especially for script-diverse and low-resource settings.

Supported Languages (11)

Model Details

• Model type: fastText supervised classifier
• Architecture: Character n-grams (2–5)
• Task: Sentence-level Language Identification (LID)
• Training data: 22,000 sentences (2,000 per language)
• Train / Dev / Test split: 70% / 15% / 15% (stratified)
• Evaluation accuracy: 99.09% (macro-F1: 0.99)
• Model size: ~10 MB
• Inference speed: <5ms per sentence

Why fastText?

Extensive experiments show that character-level models outperform transformer-based language models (e.g., NE-BERT, XLM-R) for Northeast Indian LID.

Key findings:

• Transformer models (NE-BERT, XLM-R) achieved only 9-37% accuracy on challenging samples
• fastText maintained 99%+ accuracy even on script-diverse, low-resource languages
• Character n-grams capture orthographic patterns better than subword tokenization for these languages

This model therefore prioritizes:

• ✅ Script awareness
• ✅ Orthographic cues
• ✅ Low-resource robustness

Performance

Test Accuracy: 99.09%

Benchmark Comparison

NE-LID significantly outperforms existing language identification systems on Northeast Indian languages:

Key Findings:

• NE-LID achieves 2.7× higher accuracy than the best competitor (GlotLID)
• Existing multilingual models fail to support 6-7 Northeast Indian languages
• Character n-gram approach outperforms transformer-based models for script-diverse, low-resource languages

Installation

pip install fasttext

Usage

Basic Usage (Python)

import fasttext

# Load the model
model = fasttext.load_model("ne_lid.bin")

# Predict language
text = "Ki paidbah shnong ki la ia shim bynta ha ka jingïalang"
labels, probs = model.predict(text)

print(f"Language: {labels[0].replace('__label__', '')}")
print(f"Confidence: {probs[0]:.4f}")

Output:

Language: khasi
Confidence: 0.9999

Batch Prediction

texts = [
    "Ka sngi ka lieh",
    "আজি মই বজাৰলৈ গৈছিলোঁ",
    "Mizo tawng hi a ṭha hle"
]

predictions = model.predict(texts)
for text, (label, prob) in zip(texts, zip(*predictions)):
    lang = label.replace('__label__', '')
    print(f"{text[:30]:30} → {lang:10} ({prob:.3f})")

Get Top-K Predictions

# Get top 3 language predictions
labels, probs = model.predict(text, k=3)

for label, prob in zip(labels, probs):
    lang = label.replace('__label__', '')
    print(f"{lang}: {prob:.4f}")

Limitations

• Designed for monolingual sentences – not optimized for code-mixed text
• Sentence-level only – not designed for word-level or document-level LID
• Performance may degrade on extremely short inputs (≤2 tokens)
• English/Hindi confusion at 96-97% (expected due to loanwords and script overlap)

Model Files

• ne_lid.bin - Main fastText model (binary format)
• ne_lid.ftz - Compressed model (optional, for smaller deployments)

Training Details

Data Sources:

• Training corpus derived from NE-BERT dataset
• 2,000 sentences per language, stratified by length and script
• Balanced across language families (Austroasiatic, Tibeto-Burman, Indo-Aryan)

Hyperparameters:

• Learning rate: 0.1
• Epochs: 25
• Word n-grams: 1-3
• Character n-grams: 2-5
• Loss function: Softmax

License

This model is released under Creative Commons Attribution 4.0 International (CC BY 4.0).

You are free to:

• ✅ Share — copy and redistribute the material
• ✅ Adapt — remix, transform, and build upon the material

Under the following terms:

• Attribution — You must give appropriate credit to MWire Labs

Citation

If you use NE-LID in your research or applications, please cite:

@inproceedings{nyalang-2026-ne,
  title = {NE-LID: A Fast and Accurate Language Identification System for Northeast Indian Languages},
  author = {Nyalang, Badal},
  booktitle = {Proceedings of the 8th Workshop on Indian Language Data: Resources and Evaluation},
  year = {2026},
  pages = {104--108},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  doi = {10.63317/2q69fiyg73p6}
}

About MWire Labs

MWire Labs is an AI research organization based in Shillong, Meghalaya, India, specializing in language technology for Northeast India's indigenous languages.

Repository: MWirelabs/ne-lid
Contact: MWire Labs

Acknowledgments

We thank the open-source community and contributors to the NE-BERT corpus that made this work possible.

Last Updated: January 2026
Version: 1.0.0