⭐ GLiClass Model Router

Developed by SCX in collaboration with Knowledgator.

A lightweight GLiClass-based model router that scores which of 8 candidate LLMs should handle a request, its task type, difficulty, reasoning need, expected output length, and whether a response looks hallucinated. Additionally, the model supports zero-shot classification and works naturally with chat-like inputs.

Why use it

• No generation, one pass — a calibrated probability per label, not a decoded token stream.
• Zero-shot labels — swap the model roster, task taxonomy, or add a custom label set at inference time, no retraining required.
• Streaming-native — a decoder KV cache per session means a multi-turn conversation is re-routed every turn at the cost of encoding only the new tokens, not the whole history.
• One checkpoint, many signals — model routing, 28-way task type, difficulty, reasoning mode, output length, and hallucination detection all come from the same ~0.6B model.
• Small footprint — cheap enough to co-locate with a gateway or run alongside the models it's routing to.

Installation

pip install gliclass -U

Usage

Single input

One text, scored against a label set in a single forward pass — the same call pattern for any label family (model routing, task type, difficulty, reasoning mode, output length, hallucination, or a custom label set).

from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

model_path = "scx-admin/scx-router-v0.1"  # or a local path to this checkpoint

model = GLiClassModel.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

pipeline = ZeroShotClassificationPipeline(
    model, tokenizer, classification_type="multi-label", device="cuda:0"
)

text = "Write a Python function that merges two sorted linked lists."
labels = [
    "coder", "DeepSeek-V3.1", "gemma-4-31B-it", "gpt-oss-120b",
    "Llama-4-Maverick-17B-128E-Instruct", "MAGPiE",
    "Meta-Llama-3.3-70B-Instruct", "Qwen3-32B",
]

for r in pipeline(text, labels, threshold=0.5)[0]:
    print(r["label"], "=>", round(r["score"], 3))

Streamed input — message by message

StreamingZeroShotClassificationPipeline keeps one KV cache per session_id. Each call appends only the newest message; tokens_added and cached_length show exactly what got encoded versus reused from cache.

from gliclass import GLiClassModel, StreamingZeroShotClassificationPipeline
from transformers import AutoTokenizer

model = GLiClassModel.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

router = StreamingZeroShotClassificationPipeline(
    model, tokenizer, classification_type="multi-label", device="cuda:0"
)

session_id = "chat-42"
turns = [
    "I need help refactoring some Rust code.",
    "Specifically the borrow checker keeps rejecting this function.",
    "fn parse(&mut self, buf: &[u8]) -> Result<Token, Error> { ... } -- here's the body.",
]

for turn in turns:
    out = router(turn, labels, session_ids=session_id, threshold=0.5)[0]
    print(f"+{out['tokens_added']} new tokens -> cache now {out['cached_length']} tokens")
    if out["triggered"]:
        for pred in sorted(out["predictions"], key=lambda p: -p["score"]):
            print(f"  {pred['label']:<38s} {pred['score']:.3f}")

# only each turn's own tokens are encoded -- earlier turns stay in the KV cache

Task-type classification — 28-way, single-label:

task_types = [
    "analysis", "classification", "clustering", "code", "comparison", "critique",
    "decision support", "evaluation", "explanation", "fact checking", "forecasting",
    "generation", "information extraction", "information retrieval", "instruction following",
    "math & reasoning", "multi-turn", "planning", "problem solving", "programming", "qa",
    "reasoning", "recommendation", "rewriting", "summarization", "translation",
    "verification", "visualization",
]

pipeline = ZeroShotClassificationPipeline(model, tokenizer, classification_type="single-label", device="cuda:0")
result = pipeline("Compare these two vendor contracts and flag the riskier clauses.", task_types)[0]
print(result[0]["label"], result[0]["score"])  # -> "comparison" 0.9xx

Difficulty & expected output length — feeds token-budget decisions:

difficulty_labels = ["very easy", "easy", "medium", "hard", "extra hard"]
length_labels = [
    "1-64 output tokens", "65-128 output tokens", "129-256 output tokens",
    "257-512 output tokens", "513-1024 output tokens", "1025-2048 output tokens",
    "2049-4095 output tokens",
]

text = "Derive the closed-form solution for this second-order linear recurrence and prove convergence."
difficulty = pipeline(text, difficulty_labels)[0][0]
length = pipeline(text, length_labels)[0][0]
print(difficulty["label"], length["label"])

Reasoning-mode routing — toggle extended thinking on/off:

reasoning_labels = ["reasoning", "nonreasoning"]
mode = pipeline("What's a good one-line commit message for this typo fix?", reasoning_labels)[0][0]
enable_thinking = mode["label"] == "reasoning"

Hallucination detection — score a candidate response, not the request. Treat this as a directional signal (F1 ≈ 65%, see benchmarks below) rather than a sole filtering gate:

hallucination_labels = ["correct", "hallucinated"]
answer_text = "The Eiffel Tower was completed in 1889 for the World's Fair."
verdict = pipeline(answer_text, hallucination_labels)[0][0]
print(verdict["label"], verdict["score"])

General zero-shot classification — custom labels, not just routing. The pre-training mix also covers topics, sentiment, emotion, NLI, guardrails and toxicity, so arbitrary label sets work too:

labels = ["travel", "finance", "health", "sports", "technology"]
result = pipeline("The new mid-range EV just posted a 480-mile range on a single charge.", labels)[0]
for r in sorted(result, key=lambda x: -x["score"]):
    print(r["label"], round(r["score"], 3))

Benchmarks

Threshold 0.5 unless noted. Per-family macro F1 is the unweighted mean of the listed per-class F1 scores.

Model routing — macro F1 ≈ 75.9%

Precision is consistently high (0.74–0.84) across all 8 candidates; recall is the swing factor, favoring frequently-correct generalists (Gemma 4 31B, Llama 4 Maverick) over narrower specialists (Qwen3 32B, MAGPiE).

End-to-end gain by LiveBench dataset

Reference numbers from the most recent end-to-end routing benchmark (results/report.json, 100 tasks per subset, checkpoint models/decoder_kv/post_train) — this end-to-end benchmark has not yet been re-run for lasty, so treat it as prior context rather than a lasty-specific result. Gains are largest where the 8 candidate models disagree the most (livebench/language +26.2 pts, livebench/math +18.3 pts) and smallest where they already cluster together (livebench/data_analysis +0.1 pts).

Task-type routing — macro F1 ≈ 83.7%

28-way taxonomy, single-label, alphabetical order. Near-perfect on well-separated categories (analysis, code, IE, QA, reasoning, summarization, translation); weakest on task types that overlap conceptually with others — problem solving is frequently folded into programming/math & reasoning, and decision support/evaluation/critique overlap with each other and with verification.

Difficulty routing — macro F1 ≈ 78.9%

Most top-1 misses land one ordinal step away (hard ↔ extra hard); top-k accuracy is a fairer read than exact top-1 match for an ordinal scale like this.

Reasoning-mode routing — average F1: 89.73%

The strongest head in this checkpoint.

Output-length routing — macro F1 ≈ 78.8%

Precision drops sharply in the 1,025–2,048 token bucket — the model over-predicts this bucket, likely because it sits between the more distinctive "short answer" and "long-form generation" regimes.

Details

Model families

• Model routing — which of 8 candidate LLMs is likely to solve the request (multi-label): coder, DeepSeek-V3.1, gemma-4-31B-it, gpt-oss-120b, Llama-4-Maverick-17B-128E-Instruct, MAGPiE, Meta-Llama-3.3-70B-Instruct, Qwen3-32B.
• Task type — a 28-way taxonomy: analysis, classification, clustering, code, comparison, critique, decision support, evaluation, explanation, fact checking, forecasting, generation, information extraction, information retrieval, instruction following, math & reasoning, multi-turn, planning, problem solving, programming, qa, reasoning, recommendation, rewriting, summarization, translation, verification, visualization.
• Difficulty — 5 levels: very easy, easy, medium, hard, extra hard.
• Reasoning mode — whether extended reasoning should be enabled: reasoning vs nonreasoning.
• Expected output length — token-range buckets from 1-64 up to 2049-4095 output tokens.
• Hallucination detection — whether a response is correct or hallucinated (new head in this checkpoint).
• General zero-shot classification — the pre-training mix also covers topics, sentiment, emotion, NLI, guardrails, toxicity, and safety taxonomies, so the model remains a capable general-purpose classifier beyond routing.

A new architecture, not just a new checkpoint

Every previously released GLiClass model (gliclass-small/base/large-v1.0 and successors) uses a bidirectional uni-encoder: the whole input is re-encoded from scratch on every call. This checkpoint introduces GLiClass's first decoder-KV architecture — a causal Qwen3-0.6B backbone where the conversation lives in an ordinary decoder KV cache. In a session, each new turn only pays to encode its own tokens; labels are then re-scored against the full cached context. That single change turns a stateless classifier into a streaming router.

Labels are injected as <<LABEL>> label_text segments and scored against the text representation from the decoder. Because the text context lives in the KV cache, the model supports cheap incremental/streaming classification: in a multi-turn session, only new text tokens need to be encoded on each turn, while labels are (re-)scored against the full cached context.

How this compares to other router designs

Embedding models, encoder-only classifiers, and generative LLM routers each have a real strength. Decoder-KV is the one architecture that combines inference-time labels, a chat-aligned causal backbone, and incremental dialogue state without paying autoregressive generation cost on every routing decision.