thalovant/thalovant-m2v-intents
A static-embedding intent classifier for the Thalovant hub. It maps an
utterance to one of 215 labels of the form <skill_id>:<intent>,
exactly as ovos-m2v-pipeline registers them. The labels are the fleet's
skills and the other intents the hubs register (the persona and reading
pipelines, wikihow), from the fleet corpus and nothing else. There is no
"none of these" label: a sentence about something no skill does still gets
the nearest skill's label, often with high confidence.
Base: Jarbas/m2v-256-distiluse-base-multilingual-cased-v2, fine-tuned with model2vec's classifier trainer.
How good it is
Measured on sentences the classifier never saw: 42422 held-out rows whose text, in any language, and whose intent line are both absent from the 195173 rows an evaluation model was trained on. A split per row would score the en-GB twin of an en-US training sentence as new.
- Intent accuracy 89.3 %, skill accuracy 96.0 % across 68 languages; median language 89.5 %, 8 language(s) under 80 %.
- 0.5 % of held-out sentences are given to another
skill at 0.9 or more: what
thalovant-skillkit checkwould report about a sentence that is in fact new and the skill's own. - Languages that need a higher threshold to keep that under
1.0 %:
bg-BG0.94,et-EE0.92,eu-ES0.92,fa-IR0.94,fi-FI0.99,hi-IN0.92,ko-KR0.95,lt-LT0.92,oc-FR0.95,ro-RO0.96,sv-FI0.94,sv-SE0.94 (thresholdsintraining.json).
The published weights are then trained with the same recipe on all
237595 rows, so a skill's existing sentences are the
classifier's training data. At most 512 sentences per label and language
are trained on (the reading pipeline's English intents alone expand to
thousands); index.json keeps all 260140.
Use
from model2vec.inference import StaticModelPipeline
model = StaticModelPipeline.from_pretrained("thalovant/thalovant-m2v-intents")
model.predict(["will it rain tomorrow"])
Configure the pipeline with "mode": "classifier" and this model's path;
labels.json beside the weights lists the labels the hub may route to.
index.json names every sentence in the training corpus by a digest of its
language and text, with the labels that publish it. thalovant-skillkit check
uses it to fail a skill that publishes a sentence another skill already owns,
and asks this classifier what it makes of the rest -- without the fleet's
sentences leaving their private repositories.
Held-out results per language
False warnings: held-out sentences given to another skill at 0.9 or more. Threshold: the lowest that keeps them under 1.0 %.
| Language | Rows | Intent | Skill | False warnings | Threshold |
|---|---|---|---|---|---|
an-ES | 394 | 89.6 % | 97.5 % | 0.0 % | 0.9 |
ar-SA | 696 | 78.9 % | 94.5 % | 0.3 % | 0.9 |
ast-ES | 501 | 83.4 % | 96.0 % | 0.8 % | 0.9 |
az-AZ | 358 | 79.9 % | 90.8 % | 0.0 % | 0.9 |
bg-BG | 509 | 91.9 % | 94.9 % | 1.4 % | 0.94 |
ca-ES | 755 | 92.6 % | 96.4 % | 0.1 % | 0.9 |
cs-CZ | 634 | 86.8 % | 95.1 % | 0.3 % | 0.9 |
da-DK | 1087 | 88.5 % | 96.5 % | 0.4 % | 0.9 |
de-AT | 869 | 89.4 % | 95.9 % | 0.7 % | 0.9 |
de-CH | 869 | 89.4 % | 95.9 % | 0.7 % | 0.9 |
de-DE | 1010 | 88.7 % | 95.8 % | 0.6 % | 0.9 |
el-GR | 807 | 86.1 % | 92.7 % | 0.9 % | 0.9 |
en-AU | 816 | 96.3 % | 98.7 % | 0.2 % | 0.9 |
en-CA | 814 | 96.3 % | 98.7 % | 0.2 % | 0.9 |
en-GB | 816 | 96.3 % | 98.7 % | 0.2 % | 0.9 |
en-NZ | 816 | 96.3 % | 98.7 % | 0.2 % | 0.9 |
en-US | 1523 | 96.0 % | 99.0 % | 0.1 % | 0.9 |
es-AR | 504 | 98.0 % | 99.0 % | 0.0 % | 0.9 |
es-CO | 493 | 97.8 % | 98.8 % | 0.2 % | 0.9 |
es-ES | 700 | 96.4 % | 98.0 % | 0.7 % | 0.9 |
es-MX | 492 | 98.0 % | 99.0 % | 0.0 % | 0.9 |
es-US | 492 | 98.0 % | 99.0 % | 0.0 % | 0.9 |
et-EE | 377 | 83.0 % | 90.2 % | 1.1 % | 0.92 |
eu-ES | 425 | 76.0 % | 86.1 % | 1.4 % | 0.92 |
fa-IR | 630 | 83.5 % | 94.1 % | 1.1 % | 0.94 |
fi-FI | 388 | 66.5 % | 84.0 % | 3.1 % | 0.99 |
fr-BE | 628 | 94.8 % | 96.8 % | 0.3 % | 0.9 |
fr-CA | 624 | 94.7 % | 96.8 % | 0.3 % | 0.9 |
fr-CH | 632 | 94.8 % | 96.8 % | 0.3 % | 0.9 |
fr-FR | 1071 | 95.1 % | 98.0 % | 0.2 % | 0.9 |
gl-ES | 1130 | 97.0 % | 98.3 % | 0.4 % | 0.9 |
he-IL | 685 | 91.7 % | 96.2 % | 0.4 % | 0.9 |
hi-IN | 1538 | 83.0 % | 94.0 % | 1.1 % | 0.92 |
hr-HR | 559 | 89.5 % | 96.8 % | 0.4 % | 0.9 |
hu-HU | 602 | 86.4 % | 94.7 % | 0.7 % | 0.9 |
id-ID | 625 | 89.8 % | 96.3 % | 0.5 % | 0.9 |
it-IT | 829 | 86.9 % | 96.4 % | 0.8 % | 0.9 |
ja-JP | 719 | 63.3 % | 94.8 % | 0.4 % | 0.9 |
kab | 65 | 86.2 % | 98.5 % | 0.0 % | 0.9 |
ko-KR | 390 | 87.2 % | 92.8 % | 1.5 % | 0.95 |
lt-LT | 450 | 87.3 % | 94.2 % | 1.1 % | 0.92 |
lv-LV | 474 | 85.4 % | 92.4 % | 0.4 % | 0.9 |
ms-MY | 488 | 89.5 % | 95.9 % | 0.4 % | 0.9 |
nb-NO | 620 | 93.2 % | 98.1 % | 0.7 % | 0.9 |
nl-BE | 581 | 92.1 % | 96.2 % | 0.3 % | 0.9 |
nl-NL | 756 | 87.2 % | 95.9 % | 0.5 % | 0.9 |
oc-FR | 388 | 90.0 % | 94.8 % | 1.0 % | 0.95 |
pl-PL | 846 | 89.4 % | 95.5 % | 0.5 % | 0.9 |
pt-AO | 503 | 93.2 % | 98.6 % | 0.2 % | 0.9 |
pt-BR | 698 | 93.5 % | 98.6 % | 0.1 % | 0.9 |
pt-MZ | 503 | 93.2 % | 98.6 % | 0.2 % | 0.9 |
pt-PT | 710 | 93.5 % | 98.9 % | 0.1 % | 0.9 |
ro-RO | 479 | 84.8 % | 92.7 % | 1.2 % | 0.96 |
ru-RU | 539 | 87.8 % | 96.3 % | 0.6 % | 0.9 |
sk-SK | 456 | 92.1 % | 96.7 % | 0.2 % | 0.9 |
sl-SI | 376 | 86.2 % | 93.1 % | 0.5 % | 0.9 |
sv-FI | 539 | 90.0 % | 94.6 % | 1.1 % | 0.94 |
sv-SE | 674 | 87.8 % | 94.8 % | 1.2 % | 0.94 |
sw-KE | 396 | 72.2 % | 88.4 % | 0.2 % | 0.9 |
th-TH | 374 | 70.6 % | 80.8 % | 0.8 % | 0.9 |
tr-TR | 566 | 87.3 % | 94.7 % | 0.2 % | 0.9 |
uk-UA | 635 | 87.6 % | 96.4 % | 0.6 % | 0.9 |
vi-VN | 662 | 66.6 % | 96.5 % | 0.1 % | 0.9 |
zh-CN | 473 | 95.3 % | 96.6 % | 0.6 % | 0.9 |
zh-SG | 473 | 95.3 % | 96.6 % | 0.6 % | 0.9 |
zh-TW | 869 | 96.4 % | 98.0 % | 0.5 % | 0.9 |
Held-out results per skill
| Skill | Rows | Intent | Skill | False warnings |
|---|---|---|---|---|
ovos-common-reading-pipeline-plugin | 695 | 89.5 % | 99.7 % | 0.0 % |
ovos-persona-pipeline-plugin | 1576 | 96.6 % | 96.9 % | 0.2 % |
ovos-skill-wikihow.openvoiceos | 77 | 85.7 % | 85.7 % | 7.8 % |
thalovant-skill-alarm.thalovant | 464 | 81.7 % | 93.5 % | 1.1 % |
thalovant-skill-coming-soon.thalovant | 2810 | 98.9 % | 99.2 % | 0.1 % |
thalovant-skill-custos-query.thalovant | 245 | 83.3 % | 86.9 % | 0.4 % |
thalovant-skill-custos-shadow.thalovant | 247 | 87.0 % | 93.1 % | 1.2 % |
thalovant-skill-date-time.thalovant | 1982 | 82.7 % | 94.2 % | 0.7 % |
thalovant-skill-easter-eggs.thalovant | 266 | 73.3 % | 79.7 % | 0.8 % |
thalovant-skill-fart.thalovant | 601 | 87.0 % | 94.8 % | 1.0 % |
thalovant-skill-guide.thalovant | 2708 | 86.7 % | 93.2 % | 0.8 % |
thalovant-skill-home.thalovant | 13120 | 87.9 % | 98.4 % | 0.2 % |
thalovant-skill-interaction-modes.thalovant | 310 | 80.7 % | 92.6 % | 2.3 % |
thalovant-skill-joke-garden.thalovant | 754 | 78.1 % | 82.0 % | 2.5 % |
thalovant-skill-language-buddy.thalovant | 276 | 90.9 % | 90.9 % | 1.8 % |
thalovant-skill-learning-lounge.thalovant | 320 | 90.6 % | 90.6 % | 0.3 % |
thalovant-skill-local-pulse.thalovant | 240 | 92.1 % | 92.1 % | 1.7 % |
thalovant-skill-news.thalovant | 1168 | 96.2 % | 98.7 % | 0.2 % |
thalovant-skill-on-this-day.thalovant | 690 | 96.1 % | 96.2 % | 1.0 % |
thalovant-skill-ops-copilot.thalovant | 363 | 90.9 % | 90.9 % | 0.5 % |
thalovant-skill-pop-facts.thalovant | 1583 | 88.8 % | 94.6 % | 0.9 % |
thalovant-skill-pop-quiz.thalovant | 2297 | 99.9 % | 99.9 % | 0.0 % |
thalovant-skill-recall.thalovant | 1049 | 87.1 % | 94.1 % | 1.1 % |
thalovant-skill-reminder.thalovant | 415 | 87.9 % | 95.7 % | 0.0 % |
thalovant-skill-rewrite-forge.thalovant | 501 | 87.2 % | 87.2 % | 1.6 % |
thalovant-skill-safety-guide.thalovant | 318 | 83.3 % | 83.3 % | 2.2 % |
thalovant-skill-source-scout.thalovant | 377 | 81.7 % | 81.7 % | 4.0 % |
thalovant-skill-stopwatch.thalovant | 532 | 76.1 % | 93.4 % | 0.8 % |
thalovant-skill-timer.thalovant | 486 | 83.1 % | 90.5 % | 0.6 % |
thalovant-skill-twenty-questions.thalovant | 201 | 90.5 % | 90.5 % | 0.0 % |
thalovant-skill-volume.thalovant | 1053 | 79.7 % | 95.8 % | 0.7 % |
thalovant-skill-weather.thalovant | 2404 | 86.7 % | 95.4 % | 0.5 % |
thalovant-skill-what-next.thalovant | 2294 | 97.9 % | 97.9 % | 0.1 % |
What the classifier confuses
Each line is a held-out sentence whose true intent the classifier read as another intent. Two skills on one line is a collision the fleet should resolve.
thalovant-skill-home.thalovant:home.stateread asthalovant-skill-home.thalovant:home.turn.on(265): "ye o ventilador de something enzendiu" (an-ES)thalovant-skill-home.thalovant:home.turn.onread asthalovant-skill-home.thalovant:home.turn.off(238): "أضئ الضوء في something" (ar-SA)thalovant-skill-home.thalovant:home.stateread asthalovant-skill-home.thalovant:home.turn.off(193): "ye o ventilador de something amortau" (an-ES)thalovant-skill-home.thalovant:home.turn.offread asthalovant-skill-home.thalovant:home.turn.on(90): "أوقف تشغيل الضوء في something" (ar-SA)thalovant-skill-home.thalovant:home.brightnessread asthalovant-skill-home.thalovant:home.turn.off(88): "gør loftslampen mørkere" (da-DK)ovos-common-reading-pipeline-plugin:ReadContentByCollectionread asovos-common-reading-pipeline-plugin:ReadContent(60): "kunne du fortælle en something-eventyr om something" (da-DK)thalovant-skill-home.thalovant:home.stateread asthalovant-skill-weather.thalovant:temperature.weather(45): "qué temperatura fa adentro" (an-ES)thalovant-skill-home.thalovant:home.turn.offread asthalovant-skill-home.thalovant:home.state(45): "أوقف تشغيل النور في something" (ar-SA)thalovant-skill-home.thalovant:home.turn.onread asthalovant-skill-home.thalovant:home.brightness(38): "upali svjetlo u something" (hr-HR)thalovant-skill-home.thalovant:home.brightnessread asthalovant-skill-home.thalovant:home.turn.on(37): "laita something valo kirkkaammalle" (fi-FI)thalovant-skill-recall.thalovant:did.i.askread asthalovant-skill-recall.thalovant:what.did.i.ask(36): "heb ik je al naar something gevraagd" (nl-NL)thalovant-skill-home.thalovant:home.stateread asthalovant-skill-home.thalovant:home.lock(35): "giriş qapısı kilidlidir mi" (az-AZ)thalovant-skill-home.thalovant:home.cover.closeread asthalovant-skill-home.thalovant:home.cover.open(34): "أنزل ستائر something" (ar-SA)thalovant-skill-home.thalovant:home.stateread asthalovant-skill-home.thalovant:home.cover.close(34): "ye zarrau o garaje" (an-ES)thalovant-skill-recall.thalovant:what.was.the.last.topicread asthalovant-skill-source-scout.thalovant:source.scout(33): "आपने जो पिछला something सुनाई थी वह क्या था" (hi-IN)
Trained from the corpus built 2026-10-11T23:45:12Z covering 33 skills and sources, expanded by thalovant-skillkit 0.25.0.