Eva Customer Support Manager — merged model
Qwen/Qwen3.5-4B adapted for customer-support triage in a women's health app
(cycle tracking, pregnancy, teen mode) operating in Kazakhstan in Russian and
Kazakh. A QLoRA adapter was trained separately and merged into the unquantized
half-precision base.
Given one customer message, the model emits only a JSON object classifying it on four axes. It does not write prose and is not meant to: in the system it was built for, the JSON selects a reply from an approved template library.
Intended use
Triage and routing. Downstream of this model sits a deterministic safety layer that can only reduce the system's autonomy, never raise it. Do not deploy the model without that layer. It is not a medical device, not a diagnostic tool, and must never answer a medical question on its own.
Output format
{
"category": "auth_otp",
"priority": "high",
"sentiment": "negative",
"action": "answer_from_kb"
}
| Field | Allowed values |
|---|---|
category | auth_otp · auth_account · age_guardian · geo_availability · cycle_predictions · pregnancy · health_diary · app_technical · store_order · store_payment · store_promo · store_product · medical_concern · other |
priority | low · medium · high · critical |
sentiment | positive · neutral · negative |
action | answer_from_kb · explain_expected_behaviour · request_details · create_bug_ticket · escalate_human · escalate_medical · escalate_legal_privacy |
The label space was frozen after measuring 214 real support conversations and must not be extended: the baseline and the fine-tuned model are scored against an identical set, or the comparison below is void.
Training
| Method | QLoRA — base loaded in 4-bit NF4, adapter trained, then merged into float16 |
| LoRA | r=16, alpha=32, dropout=0.05, target_modules="all-linear" |
| Schedule | 3 epochs, lr 2e-4, batch 1 × grad-accum 8, max length 1024, seed 42 |
| Final validation | eval_loss 0.560, mean_token_accuracy 0.900 |
| Merge dtype | float16 (not quantized; GGUF conversion is a separate step) |
| Split | train 629 · validation 207 · test 215 |
The test split was not used during training.
Training data
Synthetic. The corpus was generated from a catalogue of the app's real defects and a catalogue of behaviours that look like defects but are not — for example an empty "Today" screen after childbirth, or an auto-cancelled unpaid order. That second catalogue is the point: an off-the-shelf model confidently files false bug reports against intended behaviour, and no public corpus contains this knowledge.
Real customer messages were never used for training. They exist only as a held-out slice for measuring behaviour on live traffic.
Evaluation
215 held-out examples. Base model is Qwen/Qwen3.5-4B zero-shot with the same
prompt.
| Metric | Base | This model |
|---|---|---|
| Category accuracy | 0.00 % | 82.79 % |
| Action accuracy | 0.00 % | 69.77 % |
| Priority accuracy | 7.44 % | 59.53 % |
| Sentiment accuracy | 7.44 % | 56.28 % |
| Exact match, all four fields | 0.00 % | 35.81 % |
| Invalid JSON rate | 86.05 % | 0.00 % |
Read these numbers with two caveats
- The base model mostly failed on format, not on understanding: 185 of its 215 answers were invalid JSON. A large share of the category gain is format compliance. Reporting the category number without this line overstates what was learned.
- The test split is synthetic. A separate slice of 282 hand-labelled real conversations exists and has not been scored yet. Until it is, nothing here is evidence about live traffic.
Class coverage in real data
Four of the fourteen categories have no real examples at all (pregnancy,
health_diary) or exactly one (auth_account, age_guardian — the latter
safety-critical). Per-class claims about those four are not supportable, and a
macro average that hides them would be misleading.
Limitations
- Educational prototype, not a production system and not a medical device.
- Trained on synthetic data; measured on synthetic data.
- Kazakh is under-represented: 18 of 214 real conversations. The system it belongs to never grants Kazakh messages autonomous replies for this reason.
- Deidentification tooling for Kazakh does not exist, and for Russian the only benchmark was machine-translated. Do not assume input is safely scrubbed.
- Prompt injection is bounded by architecture, not by the model: the agent selects from approved templates, so the worst case is a wrong template rather than arbitrary text. Remove the template layer and that bound is gone.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "AygerimTan/eva-qwen35-4b-support-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
dtype="auto",
)
License
Inherits the base model's licence. Owner: confirm Qwen/Qwen3.5-4B's terms
and add the matching license: field to the front matter before treating this
card as final. It is deliberately left out rather than guessed.