fastText lid.176 for Scribe SV
An unmodified, byte-for-byte copy of the official fastText language identification model
lid.176.bin, published here so that Scribe SV — a Windows
dictation and translation utility — can download it on demand from a repository we control.
Nothing was retrained, quantized, converted or otherwise changed.
- Title: fastText lid.176 (language identification, 176 languages)
- Authors: Facebook AI Research — Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov et al.
- Original: https://fasttext.cc/docs/en/language-identification.html,
downloaded from
https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin - License: Creative Commons Attribution-ShareAlike 3.0 (as stated by the original publisher). This copy keeps the same license; no additional terms or restrictions are applied.
| File | Size | MD5 | SHA-256 |
|---|---|---|---|
lid.176.bin | 131 266 198 bytes | 01810bc59c6a3d2b79c79e6336612f65 | 7e69ec5451bc261cc7844e49e4792a85d7f09c06789ec800fc4a44aec362764e |
What Scribe SV uses it for
The translator works on a pair of languages chosen by the user (for example German ↔ French). For a piece of text the application asks one question: which of the two languages of the pair is this text in? It reads the full probability vector and picks the more probable of the two pair languages; the model's top-1 label is not used.
Usage
import fasttext # pip install fasttext-predict
from huggingface_hub import hf_hub_download
model = fasttext.load_model(hf_hub_download("crash-sv/scribe-fasttext-lid176", "lid.176.bin"))
labels, probs = model.predict("Guten Morgen, wie geht es dir?", k=-1)
Citation
@article{joulin2016bag,
title={Bag of Tricks for Efficient Text Classification},
author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
journal={arXiv preprint arXiv:1607.01759},
year={2016}
}
@article{joulin2016fasttext,
title={FastText.zip: Compressing text classification models},
author={Joulin, Armand and Grave, Edouard and Douze, Matthijs and J{\'e}gou, H{\'e}rve and Mikolov, Tomas},
journal={arXiv preprint arXiv:1612.03651},
year={2016}
}
Attribution and license
fastText language identification model lid.176.bin
- Author: Facebook AI Research (A. Joulin, E. Grave, P. Bojanowski, T. Mikolov)
- License: CC-BY-SA-3.0 — https://creativecommons.org/licenses/by-sa/3.0/
- Upstream: https://fasttext.cc/docs/en/language-identification.html
- Changes: Unchanged byte-for-byte copy of lid.176.bin from dl.fbaipublicfiles.com. Not quantized, not retrained.
License text (LICENSE-CC-BY-SA-3.0.txt) lies in the repository root.
This republication keeps the original licenses.