AlphaRoute-0.8B-v1.5

AlphaRoute-0.8B-v1.5 is an instruction-conditioned Small Language Model (SLM) engineered specifically for high-speed semantic routing, dynamic ontology classification, and constrained multi-field JSON extraction.

Trained on a curated 17,898-row instruction blend, AlphaRoute v1.5 learns the meta-task of routing: it does not have hardcoded static labels. At runtime, developers supply an arbitrary task instruction, a dynamic ontology (with descriptions), and an arbitrary JSON schema; AlphaRoute outputs valid JSON matching the requested schema at edge-level latencies.

Benchmark Evaluation (Deterministic Fixed Splits)

All evaluations are conducted with greedy decoding (temperature = 0.0) on standardized fixed test splits:

Understanding the CLINC150 Out-of-Scope (OOS) Recall: Trade-off vs. Regression

At initial glance, AlphaRoute v1.5 records an 85.11% OOS recall on CLINC150, whereas earlier models and frontier APIs scored between 94.68% and 98.94%. A granular breakdown of the 500 benchmark episodes (406 In-Scope user requests vs. 94 Out-of-Scope queries) clarifies why this reflects a deliberate optimization trade-off rather than an architectural regression:

Key Factors Behind This Dynamic:

1. Elimination of False-Rejection Bias:

   • Earlier models achieved high OOS recall by exhibiting an overly defensive bias: when faced with slight semantic ambiguity, they defaulted to out_of_scope: true.
   • Consequently, DeepSeek misclassified or discarded 35.0% of all legitimate user requests (142 real queries falsely rejected), and v1.1 discarded 32.3%.
   • AlphaRoute v1.5 resolved this bias, surging in-scope accuracy from 67.73% to 96.06% (390/406 valid requests correctly routed). This lifted overall CLINC150 accuracy from 73.20% to 94.00% (+20.80% gain).

2. Lexical Proximity in 10-Category Sampling:

   • In the CLINC150 evaluation setup, each query is evaluated against 10 categories sampled from 150 broad conversational intents.
   • Dissection of the 14 unflagged OOS queries revealed that random sampling frequently paired an out-of-scope query with an intent sharing literal keywords (e.g., "what oil is best for chicken" paired with oil_change_how, or "distance between football goal posts" paired with distance). Because v1.5 is trained to route confidently rather than defensively drop requests, it selected the overlapping label.

3. Enterprise Boundary Precision:

   • On production-style ontologies with clear operational scopes (such as the Adversarial OOS benchmark suite and Golden 300 Enterprise), AlphaRoute v1.5 achieves 100.00% out-of-scope rejection.

Evolutionary Lineage and Design Methodology

AlphaRoute-0.8B v1.5 was developed through a structured, multi-phase research progression:

1. AlphaRoute-0.8B v1.0 (Baseline Exploration):

   • Established initial proof-of-concept for instruction-conditioned semantic routing on a sub-billion parameter model.
   • Highlighted key behavioral limitations: 0.0% compound accuracy on multi-target nested schemas, susceptibility to keyword distractors (35.0% adversarial out-of-scope accuracy), and inconsistent priority grading.

2. AlphaRoute-0.8B v1.1 (Internal Research Iteration):

   • Rather than continuing from the v1.0 checkpoint, v1.1 was a fresh fine-tune of the base model (Qwen/Qwen3.5-0.8B-Base) designed specifically to address the core behavioral issues identified in v1.0.
   • Introduced multi-field schema extraction (improving compound accuracy to 52.1% and Golden 300 Enterprise routing to 95.67%).
   • Role: Served as the internal research baseline and foundation for subsequent data blend curation.

3. AlphaRoute-0.8B v1.5 (Public Release):

   • Built upon the architectural and training insights of v1.1, introducing a unified canonical prompt formatting pipeline and a curated 17,898-row instruction dataset.
   • Mitigated False-Rejection Bias: Addressed the over-rejection tendency present in earlier models and frontier APIs (which defaulted to out-of-scope on ambiguous queries), achieving 96.06% in-scope accuracy and advancing CLINC150 accuracy to 94.00%.
   • Held-Out Generalization: Achieved 89.96% on HWU64 without any exposure to this dataset during training.
   • Adversarial Precision: Reached 100.00% rejection on adversarial edge-case distractors.

Quickstart Inference

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NamanAgnih0tri/AlphaRoute-0.8B-v1.5"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

task = "Triage enterprise infrastructure incident. Disentangle primary root cause, determine responding operational team, and extract urgency."
categories = {
    "db_deadlock": "Postgres/MySQL lock contention, exclusive table locks, query timeouts.",
    "k8s_oom": "Kubernetes OOMKilled exit code 137, cgroup memory exhaustion, pod evicted.",
    "tls_cert_expired": "TLS handshake failure, expired origin cert on ALB/Cloudflare.",
    "iam_privilege_escalation": "Compromised access keys, unauthorized IAM role modification."
}
output_schema = {
    "root_cause": "db_deadlock | k8s_oom | tls_cert_expired | iam_privilege_escalation | null",
    "assigned_team": "dba_sre | k8s_platform | secops | netops",
    "severity": "SEV0_critical | SEV1_high | SEV2_medium",
    "out_of_scope": "boolean"
}
query = "CRITICAL: Checkout service is stuck in CrashLoopBackOff. Logs show: `FATAL: deadlock detected Process 8912 waiting for ExclusiveLock on relation orders`."

# Construct Canonical Routing Prompt (Zero-Shot)
cat_str = "\n".join([f"- {k}: {v}" for k, v in categories.items()])
schema_str = json.dumps(output_schema, indent=2)

prompt = f"""You are a precise semantic routing engine. Analyze the input against the provided categories and output only a valid JSON object matching the requested schema. Do not output conversational text or markdown code fences.

TASK:
{task}

CATEGORIES:
{cat_str}

OUTPUT SCHEMA:
{schema_str}

INPUT:
"{query}"

JSON:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.0,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()

Tip: Guaranteed Schema Adherence with Constrained Decoding (lm-format-enforcer)

For production environments requiring strict schema guarantees, token generation can be physically constrained at each step using a grammar/FSM parser such as lm-format-enforcer:

pip install lm-format-enforcer

from lmformatenforcer import JsonSchemaParser
from lmformatenforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn

# Formal JSON schema representing your target fields and enums
json_schema = {
    "type": "object",
    "properties": {
        "root_cause": {
            "anyOf": [
                {"type": "string", "enum": ["db_deadlock", "k8s_oom", "tls_cert_expired", "iam_privilege_escalation"]},
                {"type": "null"}
            ]
        },
        "assigned_team": {"type": "string", "enum": ["dba_sre", "k8s_platform", "secops", "netops"]},
        "severity": {"type": "string", "enum": ["SEV0_critical", "SEV1_high", "SEV2_medium"]},
        "out_of_scope": {"type": "boolean"}
    },
    "required": ["root_cause", "assigned_team", "severity", "out_of_scope"],
    "additionalProperties": False
}

parser = JsonSchemaParser(json_schema)
prefix_function = build_transformers_prefix_allowed_tokens_fn(tokenizer, parser)

# Pass prefix_allowed_tokens_fn directly into model.generate
outputs = model.generate(
    **inputs,
    prefix_allowed_tokens_fn=prefix_function,
    max_new_tokens=120,
    temperature=0.0,
    do_sample=False
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
print(response)

Constrained decoding ensures 100% JSON validity, eliminates any markdown formatting leaks, and enforces allowed enum values at runtime.

Prompt Specification and Few-Shot In-Context Guidance

AlphaRoute v1.5 is trained on a mixture of zero-shot and dynamic few-shot (1 to 3 exemplars) instruction templates.

Canonical Prompt Structure

You are a precise semantic routing engine. Analyze the input against the provided categories and output only a valid JSON object matching the requested schema. Do not output conversational text or markdown code fences.

TASK:
<task description>

CATEGORIES:
- <category_1>: <description_1>
- <category_2>: <description_2>

OUTPUT SCHEMA:
<json_schema>

[OPTIONAL] EXAMPLES:
Input: "<representative query 1>"
Output: {"target_field": "category_1", ...}
Input: "<representative query 2>"
Output: {"target_field": "category_2", ...}

INPUT:
"<runtime input query>"

JSON:

How Few-Shot Exemplars Improve Accuracy

While AlphaRoute achieves high accuracy out of the box in zero-shot mode, incorporating 1 to 3 few-shot examples is recommended for specialized edge cases:

1. Resolving Ambiguous Domain Boundaries: In complex enterprise domains where two categories share surface terminology (e.g. database connection exhaustion vs backend HTTP socket timeout), providing 1 exemplar for each category sharpens the decision boundary.
2. Multi-Target Schema Value Calibration: When extracting compound fields (such as urgency levels or specialized department queues), few-shot examples anchor the target scale and eliminate bias toward default median values.
3. Internal Acronyms and Technical Vocabulary: Exemplars allow the model to recognize company-specific abbreviations and internal naming conventions without requiring model retraining.

Operational Guidelines

• Decoding Temperature: Use temperature = 0.0 (Greedy Decoding / do_sample=False) for deterministic routing decisions and valid JSON schema fidelity.
• Batch Processing: Compatible with batch generation workflows across edge environments (Apple Silicon MPS, NVIDIA CUDA, vLLM).

Feedback and Inquiries

If you have any questions, evaluation inquiries, or feedback regarding AlphaRoute, feel free to reach out:

• LinkedIn: Naman Agnihotri