MixLID

Token-Level Language Identification in Mixed-Language Text

Demo: Hugging Face Space

Paper: Coming soon.

MixLID identifies the language of individual tokens, including short expressions embedded in another language. It supports 111 language labels and returns both language spans and one predominant language per input. Numbers, punctuation, and symbols have separate labels: NUM, PUNCT, and SYM.

Highlights

• Identify languages within mixed text: locate language spans, including short expressions embedded in another language, across 111 supported language labels.
• Two prediction modes: identify the language of individual tokens or the predominant language of an entire input with the same model.

Potential applications

Usage

pip install torch==2.9.1 transformers==4.57.0 safetensors==0.7.0

Load the downloaded model directory, or replace "./MixLID" with the published Hugging Face model ID:

from transformers import AutoModel

model = AutoModel.from_pretrained("./MixLID", trust_remote_code=True)
# To use a GPU: model = model.to("cuda:0")

text = "The exhibition brings together photographs and handwritten letters."
sentence = model.predict_sentence(text)
print(sentence["language"], sentence["score"])

result = model.predict_tokens(text)
for span in result["spans"]:
    print(span["text"], span["label"], span["start"], span["end"], span["score"])

trust_remote_code=True loads the prediction methods included in this repository. When using a Hub model ID, Transformers downloads the required Python files automatically.

Download the repository, including mixlid.py, and run Python from that directory:

from mixlid import MixLID

model = MixLID.from_pretrained()  # Loads the checkpoint beside mixlid.py.
# To use a GPU: model = MixLID.from_pretrained(device="cuda:0")

text = "The exhibition brings together photographs and handwritten letters."
sentence = model.predict_sentence(text)
print(sentence["language"], sentence["score"])

result = model.predict_tokens(text)
for span in result["spans"]:
    print(span["text"], span["label"], span["start"], span["end"], span["score"])

This method loads the built-in Transformers architecture without enabling remote code. Both interfaces provide the same predictions: predict_sentence() returns the predominant language of each input, and predict_tokens() returns subword predictions and merged language spans. Both accept a string or a list of strings. predict_sentence() does not split paragraphs into sentences.

Examples

The outputs below were checked with the released checkpoint.

Korean, English, and French

text = '연구팀은 대학원생들과 함께 1,000개 문서를 검토했고, the initial results를 바탕으로 오류 유형을 정리했다. 프랑스 측은 «Nous allons poursuivre cette recherche avec une équipe internationale.»라고 밝혔다. 평균 점수는 3.14였으며, 다음 분석에서는 a larger sample을 사용할 예정이다.'

result = model.predict_tokens(text)
result["spans"]  # Each span contains text, start, end, label, and score.

Turkish, German, and English

text = 'Yeni sergiye nach der Arbeit gitmeyi düşünüyoruz; girişte free admission yazıyor ama atölyeler için eine Anmeldung gerekiyor. Çocuklara yönelik bölümde hands-on activities var, bu yüzden mit der ganzen Familie gitmek daha keyifli olabilir.'

result = model.predict_tokens(text)
result["spans"]  # Each span contains text, start, end, label, and score.

Performance

Monolingual identification

Each baseline and MixLID are evaluated on their shared language set; #L gives its label count. ACC is overall accuracy, and Macro F1 averages per-language F1. Scores are percentages, with the higher score in each comparison in bold. Comparison sets differ across baselines.

For fastText lid.176, CommonLingua, and Polyglot Tagger, script eligibility follows the evaluation mapping where explicit official script coverage is unavailable.

Mixed-language identification

Each cell reports ACC / Macro F1 (%). All models are evaluated on the same eligible tokens. Scores are computed per example and averaged within each language pair; Avg is the arithmetic mean across pairs. Non-language, mixed, and ambiguous reference labels, and tokens consisting only of whitespace, numbers, punctuation, or symbols, are excluded consistently.

ACS

CodeMixBench

Output format

• Tokens: subword predictions with text, start, end, label, and score.
• Spans: consecutive predictions with the same label, merged across intervening ordinary spaces. These are language spans, not word segmentation.
• Input language: majority vote over retained subword language predictions, excluding NUM, PUNCT, and SYM. Its score is the winning vote share, not a calibrated probability.
• Offsets: zero-based character positions in the original Python string; end is exclusive. Input text is not normalized.

Special tokens, padding, and ordinary space-only ranges are omitted. Tabs and newlines retain model predictions. Empty input or input without language votes returns und with score 0; this is a helper fallback, not a classifier label. Token scores are softmax probabilities, and span scores average the constituent token scores.

Supported languages

MixLID predicts 111 language labels, listed below with their full names. Labels distinguish scripts where applicable; for example, Mandarin Chinese has separate simplified and traditional Chinese labels. The additional labels are NUM (numbers), PUNCT (punctuation), and SYM (symbols).