laya-fa-support

Fine-tuned by Ali Jahani · Telegram: t.me/tarfandoonchannel

A Persian customer-support triage model, fine-tuned from Laya's laya-multilingual checkpoint. It reads a Persian support message and decides which department should handle it (billing, technical, cancel or other) in a single forward pass. It returns a probability for every option and generates no text, so there is nothing to parse and nothing to hallucinate.

Results at a glance

These scores come from the held-out persian_fa benchmark: 64 Persian support messages the model never saw during training, asked with prompts it never saw during training.

Quick start

pip install laya

import laya

agent = laya.load("tarfandoon/laya-fa-support")            # add device="cuda" for a GPU

question = {
    "department": {
        "type": "choice",
        "instructions": "این پیام پشتیبانی باید به کدام بخش ارجاع داده شود؟",
        "criteria": {
            "billing": "پرداخت، کسر وجه، استرداد پول، فاکتور، قیمت و هزینه‌ی اشتراک",
            "technical": "خطا، باگ، کرش، مشکل ورود یا رمز، کندی و قطعی برنامه",
            "cancel": "کاربر می‌خواهد اشتراک را لغو کند، تمدید را متوقف کند یا حساب را ببندد یا حذف کند",
            "other": "هر چیز دیگر: تشکر، پیشنهاد، همکاری، استخدام، سؤال عمومی",
        },
    }
}

result = agent.predict({"message": "سلام، دو بار پول از حسابم کم شده ولی اشتراکم فعال نشده"}, question)
answer = result["answers"]["department"]
print(answer["choice"])              # one of: billing, technical, cancel, other
print(answer["probabilities"])       # probability for every department
print(answer["answer_confidence"])   # calibrated probability of the chosen answer

Many messages at once

messages = ["اپ باز نمیشه", "mikham eshterakam ro laghv konam", "مرسی از پشتیبانی‌تون"]
results = agent.predict_batch([{"message": m} for m in messages], question)
for m, r in zip(messages, results):
    print(r["answers"]["department"]["choice"], "<-", m)

Send uncertain cases to a human

THRESHOLD = 0.6   # choose this on your own labelled data
if answer["answer_confidence"] >= THRESHOLD:
    route_to(answer["choice"])
else:
    send_to_human(answer)

Tips

• Keep the four option keys (billing, technical, cancel, other) and give every option a full description, like the example above. The wording can change, because the model was trained on several paraphrases in Persian and English. Persian instructions work best for this model (see the results).
• ⚠️ Do not use one-word descriptions such as "billing": "مالی". In a spot check on 8 new messages, the full prompt above got 7/8 right, while one-word descriptions got 5/8. The one-word version was also confidently wrong: it labelled «اپ باز نمیشه» as cancel with 0.92 confidence.
• The input can be formal or colloquial Persian, Finglish, or Persian mixed with English words. Arabic ي/ك and missing half-spaces are handled.
• On a CPU a decision takes roughly 150-200 ms. On a GPU it is much faster.

Full benchmark results: before vs after

Base model: convaiinnovations/laya / multilingual @ 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851. Scores are from the first of three repeats, and every repeat gave identical predictions. Neither model saw these 64 messages or these two prompts during training.

Persian instructions (choice_fa)

Base

This model

English instructions (choice_en)

Base

This model

Training

The dev set has 40 messages x 4 training prompts. Dev accuracy went from 0.562 before training to 0.700 at the selected epoch.

A near-duplicate check (character 3-gram Jaccard < 0.5) guarantees that no training or dev message copies a benchmark message.

Limitations

• The data is small and synthetic. The training set and the benchmark were drafted the same way, with AI assistance. Real customer traffic will be harder, so expect a smaller gain there than the benchmark shows. Evaluate on your own messages before relying on the model.
• With Persian instructions, these families did not improve: finglish (4/8).
• With English instructions, these families did not improve: colloquial (6/8), orthography (5/8), code_mixed (6/8), sarcasm_taarof (5/8), digits (5/8).
• With English instructions the overall change is small (+3 of 64), which is within noise for 64 cases.
• It is sensitive to how the options are described. Use descriptive option texts (see Tips). Terse one-word options made it much worse, with high confidence.
• It is specialised to one 4-way decision. Other Laya tasks and languages were not re-evaluated and may have degraded. Use the original Laya checkpoints for those.
• Confidence was calibrated on 160 dev rows only. Refit the threshold or temperature on your own data before gating decisions on it.

Links

• 📊 Benchmark (data, code, all raw results): https://github.com/alipyth/laya-persian-benchmark
• 🛠️ Training code and data: https://github.com/alipyth/laya_fine_tune_fa
• 🧠 Laya: https://github.com/NandhaKishorM/laya (Apache-2.0)
• 📣 Ali Jahani: https://t.me/tarfandoonchannel

License

Apache-2.0, the same as the base Laya weights.