LID framework: trained candidate models

These are the 102 trained language identification (LID) models that populate the Meta-Knowledge Base of the LID model selection framework (GitHub: Whugo226/lid_framework), developed for the MEng thesis An Interactive Framework for Language Identification Model Recommendation (Werner Hugo, Stellenbosch University, 2026).

The framework profiles an unlabelled short-text corpus and recommends one of these models (or one of three off-the-shelf detectors). Its dashboard downloads only the model it is asked to run, so you do not need to download this whole repository.

Contents

Six configurations were trained on each of 17 benchmark corpora (6 × 17 = 102 models):

Files are laid out as {family}/{training_corpus}/{configuration}/, where {family} is naive_bayes, logistic_regression or fasttext. Each folder also holds run_metadata.json (selected hyperparameters and training-split metrics), classification_report.txt, and the hyperparameter search results.

Every model predicts ISO 639-1 codes for the subset of the 24 languages below that its training corpus contains. The FastText models are large (up to about 2.3 GB each); the Naive Bayes and logistic regression pipelines are 1–63 MB.

Usage

With the framework installed, the dashboard's Run model tab downloads and runs a model on demand. Programmatically:

from lid_toolkit.explainer.model_runner import ModelRunner

# store_datasets: the training-corpus names the variant names may end in
runner = ModelRunner(store_datasets=["xnli"])
runner.predict("fasttext_subword_xnli", ["Dit is een korte zin."])  # downloads on first use

Without the framework, the files load directly:

from huggingface_hub import hf_hub_download
import fasttext, pickle

ft = fasttext.load_model(hf_hub_download(
    "werner1hugo/lid-framework-models", "fasttext/xnli/fasttext_subword/model.bin"))
ft.predict("Dit is een korte zin.")

with open(hf_hub_download(
        "werner1hugo/lid-framework-models",
        "logistic_regression/xnli/tfidf_lr_char_ngram_3_5/best_pipeline.pkl"), "rb") as fh:
    lr = pickle.load(fh)
lr.predict(["Dit is een korte zin."])

The scikit-learn pipelines were saved with scikit-learn 1.8.0. The framework pins 1.7.2 (the version its Meta-Knowledge Base was built with); under 1.7.2 the pipelines load and predict correctly but scikit-learn prints an InconsistentVersionWarning. Only unpickle files from sources you trust.

Training

• Every source document was truncated at a word boundary to at most 120 characters (40 for Chinese, Japanese and Korean); documents shorter than 30 characters (10 for CJK) were discarded.
• Each corpus was split into four disjoint row-level parts under a fixed seed: 50% knowledge-training, 20% knowledge-benchmarking, 15% evaluation and 15% validation. Models were trained on the knowledge-training split only.
• Hyperparameters were selected by macro-F1 on a stratified 80/20 split of the training data: randomised search with 5-fold (Naive Bayes) or 3-fold (logistic regression) cross-validation, and a grid over epochs and learning rate on a 90/10 split for FastText.

Performance and limitations

In-domain (evaluated on the same corpus they were trained on), mean weighted F1 ranges from 0.939 (FastText word) to 0.995 (FastText subword). Cross-domain performance collapses to a mean of about 0.55–0.61, with worst cases below 0.06: a model trained on one corpus can be unusable on another. That is why the framework recommends a model per corpus rather than one model for everything.

• Only 24 high-resource languages are covered; any other language is misclassified.
• The models were trained and evaluated on short text (30–120 characters).
• They assign one language per document and do not handle code-switching.

Training corpora and licences

These models are derived from the corpora below. They are released under CC BY-NC-SA 4.0 (non-commercial, share-alike), the most restrictive terms among the corpora with stated licences. Use of an individual model is also subject to the terms of the corpus it was trained on. Licences are as stated on each dataset card at the time of release; "not stated" means the card gives none.

Languages counts the in-scope languages each corpus contains after filtering to the 24 supported languages.

Citation

If you use these models, please cite the thesis and the software archive (see CITATION.cff in the GitHub repository).