TeleQnA quality estimator for Gemma4-E4B
Binary quality estimator over Gemma4-E4B answers on TeleQnA. Given a question
and the model's own answer, it predicts whether that answer is correct
(accept) or should be escalated to a stronger model (route).
Companion to the Qwen3-4B-Instruct estimator. A quality estimator reads the efficient model's own output, so it is specific to that model; on a single A100 the Stage 1 routing selects Gemma4-E4B where a two-card configuration selected Qwen3-4B-Instruct.
Results
Evaluated on run 0 of the TeleQnA test split, 1,000 questions.
| precision | recall | F1 | support | |
|---|---|---|---|---|
| Route | 0.54 | 0.46 | 0.50 | 321 |
| Accept | 0.76 | 0.82 | 0.79 | 679 |
| accuracy | 0.700 | 1000 | ||
| macro avg | 0.65 | 0.64 | 0.64 |
Predicting accept for everything would give accuracy 0.679 and macro F1 0.404,
so the gain is real but moderate. Route recall of 0.46 means it misses over half
the answers that should escalate.
Matched against the Qwen3-4B-Instruct estimator, trained identically:
| Qwen3-4B-Instruct | Gemma4-E4B | |
|---|---|---|
| accuracy | 0.698 | 0.700 |
| F1 macro | 0.635 | 0.638 |
Every figure agrees within 0.005, so swapping the efficient tier costs nothing in estimator quality; the ceiling is a property of the task.
Training
cre qe-train from CRE-Router, on
ModernBERT-base: 10 epochs, learning rate 2e-5, max length 512, batch 64, early
stopping patience 4, class weights on. Trained on all five runs of the train
split (45,000 rows) with run 0 of the test split for early stopping, matching the
Qwen3-4B-Instruct estimator exactly so the two remain comparable.
Generations came from gemma-4-E4B-it under vLLM 0.19.0 on 1 x A100 80GB at
concurrency 32.