XLM-V-Tuktuk-HIEN-LID
Token-level language identification for code-mixed Hindi-English (Hinglish) social text, fine-tuned from facebook/xlm-v-base.
Model Description
XLM-V-Tuktuk-HIEN-LID classifies each token in Roman-script Hinglish text into one of four classes: EN (English), HI (Hindi), NE (Named Entity), or OT (Other). Sentence-level LID systematically fails on code-switched text — one embedded English word inside a Hindi sentence gets swallowed into a single majority-vote label — so this model operates at the token level instead.
- Developed by: Ansh Sharma
- Model type: Token classification (XLM-V / XLM-RoBERTa-style encoder)
- Languages: English (
en), Hindi (hi), code-switched Hinglish - License: Apache 2.0
- Finetuned from:
facebook/xlm-v-base - Repository (code, scrapers, preprocessing): https://github.com/ansh2903/XLM-V-Tuktuk-HIEN-LID
- Full write-up (methodology, failure modes, postmortem): https://ansh2903.github.io/Ansh-Portfolio/Exp_02_LID_Fine-tuning.html
Intended Use
Designed as an early-pipeline layer feeding downstream NLP on Hinglish social text — e.g. routing tokens to the correct toxicity/sentiment lexicon, segmenting code-mixed text before machine translation, or filtering which tokens a monolingual NER/POS tagger should see. Not intended for sentence-level classification, formal/edited text, or code-switched pairs other than Hindi-English.
Training Data
~30k rows of Roman-script Hinglish comments self-collected from Reddit, Twitter/X, Facebook, Instagram, and YouTube, cleaned down to ~28,333 rows (deduplication, length filtering, non-Hinglish removal). Ground truth was produced via weak supervision — a 3-model majority-vote ensemble (Fleiss' κ ≈ 0.65) — rather than human annotation, given no annotation budget or team. Full labeling methodology and known caveats are documented in the write-up linked above.
Split: ~27,333 rows fine-tuning / 1,000-row holdout (random_state=42, labels excluded from training).
Training Procedure
| Setting | Value |
|---|---|
| Base model | facebook/xlm-v-base (encoder) |
| Fine-tuning | Full-parameter |
| Hardware | 1× Kaggle T4 GPU |
| Optimizer | adamw_8bit |
| Learning rate | 2e-5, 100 warmup steps |
| Epochs | 3 |
| Effective batch size | 32 (micro-batch 1 × 32 gradient accumulation steps) |
| Precision | fp32 |
| Training time | ~2.4 hours |
An initial attempt fine-tuned Qwen3.5-0.8B (a decoder LM) on the same data; it hallucinated at test time and failed to generalize to holdout. Switched to XLM-V — an encoder architecturally suited to token classification — for this checkpoint.
Evaluation
Important limitation: the holdout set shares the same pseudo-labeling pathway as the training data, so it measures agreement with the labeling ensemble, not independent generalization. The numbers below on the external LINCE benchmark (OT/NE collapsed to a 3-class en/hi/rest schema for comparability) are what any real performance claim should rest on.
External benchmark — LINCE (cleaned tokens)
| Model | Accuracy | Weighted F1 | Macro F1 | OT F1 |
|---|---|---|---|---|
| XLM-V-Tuktuk-HIEN-LID | 0.91 | 0.91 | 0.87 | 0.72 |
| XLM-R-RoBERTa-HIEN-LID | 0.92 | 0.91 | 0.88 | 0.76 |
| codeswitch-hineng-LID-Lince | 0.90 | 0.89 | 0.85 | 0.67 |
| HingBERT-LID | 0.80 | 0.73 | 0.58 | 0.00 |
Tuktuk sits close to XLM-R-RoBERTa-HIEN-LID (a larger, specialized model) on macro F1 — a modest, competitive result on external data, not a zero-shot win over the field.
Per-class breakdown (LINCE, cleaned)
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| EN | 0.90 | 0.97 | 0.93 | 1,370 |
| HI | 0.94 | 0.97 | 0.95 | 1,026 |
| OT | 0.86 | 0.61 | 0.72 | 467 |
Limitations
- Ground truth is weak-supervision pseudo-labels (moderate inter-model agreement, κ ≈ 0.65), not human-verified gold.
- The OT class is rare and imbalanced (<1% of tokens) and is the hardest class for every model evaluated here, including this one.
- Trained and evaluated only on Roman-script Hindi-English; not tested on Devanagari script or other code-switched language pairs.
- Confidence scores were not retained during pseudo-labeling, so voting ties were resolved by a fixed rule rather than confidence weighting (planned for a v2 pass).
Related Models & Data
- Base model:
facebook/xlm-v-base - Benchmarked against:
AryPratap/XLM-roberta-HIEN-LID,l3cube-pune/hing-bert-lid,sagorsarker/codeswitch-hineng-lid-lince - External eval set: LINCE HIEN benchmark