etomoscow/tool-calling-hallucination-modernbert-large-crf-best

Uniform soup of 6 ModernBERT-large + CRF students for character-level hallucination span detection in tool-calling LLM answers. Given a multi-turn tool-calling prompt and the model's final answer, it returns the character spans of the answer that are not grounded (hallucinated). On the s-nlp/toolace-unified-hallucinations test split (5,709 rows) it reaches Span F1 0.8854 and CharF1 0.9915, beating both the s-nlp base baseline (0.8425 / 0.9837) and the non-CRF large model (0.8685 / 0.9886).

Each student is a ModernBERT-large token classifier warm-started from etomoscow/tool-calling-hallucination-modernbert-large, fine-tuned with a linear-chain CRF head (Viterbi decoding). The 6 students are decorrelated (different weight decay / data oversampling / learning rate / epoch count), so averaging their weights yields a more robust model than any single one — the "model soup" recipe (Wortsman et al. 2022), which is weight merging, not inference-time ensembling.

Results

All numbers below are computed with the authoritative evaluate.py from s-nlp/Tool-Call-HD (branch upload-eval-script), over the full 5,709-row test split of s-nlp/toolace-unified-hallucinations. Every model is evaluated with the same build_prompt pipeline (System/User/Tool conversation format) so the comparison is apples-to-apples; our model additionally uses CRF Viterbi decoding with calibrated bias=-0.5, min_span=3.

• Span F1 (IoU≥0.75): +0.043 over the base baseline, +0.017 over the non-CRF large model.
• CharF1: +0.008 over base, +0.003 over large.
• Binary hallucination F1 ≈ 0.99 for all three (response-level detection is saturated; the differentiator is span localization).

Metric definitions (from evaluate.py): CharF1 = global micro character-overlap F1 (Σoverlap/Σpred, Σoverlap/Σgold). Span F1 (IoU≥0.75) = greedy pairwise span matching, a pair is TP if IoU≥0.75, summed globally to P/R/F1.

Cross-check: the large model was independently measured at Span F1 0.8597 by a colleague on the same test split — within ~1pt of the 0.8685 above, confirming the pipeline.

Soup ingredients (uniform weight average)

All trained: lr ∈ {1e-4, 2e-4}, 6–8 epochs, effective batch 16, cosine, warmup 0.1, bf16 + FlashAttention2 + adamw_torch_fused, CRF-NLL loss.

⚠️ Inference requires the CRF head + Viterbi (NOT plain argmax)

This model is a classifier plus a linear-chain CRF. Decode with Viterbi and apply the calibrated emission bias (-0.5) and min-span filter (3 chars). crf.py (bundled) is a dependency-free LinearCRF.

# pip install transformers torch safetensors huggingface_hub; flash-attn optional (use attn="sdpa" if unavailable)
import os, sys, torch
from transformers import AutoTokenizer, AutoModelForTokenClassification
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download

MODEL = "etomoscow/tool-calling-hallucination-modernbert-large-crf-best"
BIAS, MIN_SPAN, MAX_LEN = -0.5, 3, 4096

sys.path.insert(0, os.path.dirname(hf_hub_download(MODEL, "crf.py")))  # bundled LinearCRF
from crf import LinearCRF

tok = AutoTokenizer.from_pretrained(MODEL)
enc = AutoModelForTokenClassification.from_pretrained(
    MODEL, attn_implementation="flash_attention_2", dtype=torch.bfloat16).cuda().eval()
crf = LinearCRF(num_tags=2, batch_first=True); crf.float()
crf.load_state_dict(load_file(hf_hub_download(MODEL, "crf.safetensors"))); crf.cuda().eval()

def predict_spans(prompt: str, answer: str) -> list[dict]:
    """Return hallucinated char spans, relative to `answer`, as {'start','end'} dicts."""
    e = tok(prompt, answer, truncation="only_first", max_length=MAX_LEN,
            return_offsets_mapping=True, return_tensors="pt")
    offsets = e.pop("offset_mapping")[0].tolist()
    logits = enc(input_ids=e["input_ids"].cuda(),
                 attention_mask=e["attention_mask"].cuda()).logits[0].float()
    logits[..., 1] += BIAS                                   # emission calibration
    n_ans = len(tok(answer, add_special_tokens=False)["input_ids"])
    T = logits.shape[0]; a_start = T - n_ans - 1
    a_off = offsets[a_start][0]                             # char offset of answer start
    ans_logits = logits[a_start:a_start + n_ans].unsqueeze(0)
    mask = torch.ones(1, n_ans, dtype=torch.bool, device=logits.device)
    preds = crf.decode(ans_logits, mask)[0]                 # Viterbi over answer region
    spans, cur = [], None
    for k, p in enumerate(preds):
        ts, te = offsets[a_start + k]
        if ts == te: continue
        s, e_ = ts - a_off, te - a_off
        if p == 1:
            cur = [s, e_] if cur is None else [cur[0], e_]
        elif cur is not None:
            spans.append(cur); cur = None
    if cur is not None: spans.append(cur)
    return [{"start": s, "end": e} for s, e in spans if e - s >= MIN_SPAN]

# prompt = "System: ...\n\nUser: ...\n\nTool: ..." (everything except the final answer)
# answer  = the model's final_answer text
for sp in predict_spans(prompt, answer):
    print(sp, repr(answer[sp["start"]:sp["end"]]))

Spans are character offsets relative to answer (matching the dataset's span_labels). Build prompt from all turns except final_answer.

Evaluation protocol

• Script: evaluate/evaluate.py from s-nlp/Tool-Call-HD (branch upload-eval-script).
• Data: s-nlp/toolace-unified-hallucinations test split (5,709 rows).
• Prompt format: build_prompt (System / User / Tool turns), identical for all three models.
• Metrics: CharF1 (global micro char-overlap) and Span F1 (greedy IoU≥0.75).
• Calibration (bias/min_span) tuned on dev, never on test.

ToolHACE test results

The same published CRF soup was additionally evaluated on the current s-nlp/toolHACE test snapshot (11,606 rows; dataset revision 14c954955e445548aa882b4f65810f958dbd9d15). The table below contains test results only; no development-set numbers are reported here. Decoding used bf16, max_length=4096, emission bias -0.5, min_span=3, and no span transform. The primary metric is greedy one-to-one span F1 at IoU > 0.75.

clean and undergeneration rows have no gold hallucination spans by design; they are included in the pooled ALL score and contribute false positives when predicted spans are emitted.

Limitations

• undergeneration and clean examples have empty gold spans by dataset design; on Span F1 they contribute TN/FN as expected (not a detection failure).
• Spans are char-level offsets into the final answer; prompts >4096 tokens are left-truncated.
• Calibration (bias, min_span) was tuned on the s-nlp dev split; re-tune for other distributions.

Files

• model.safetensors — encoder + classifier head (uniform average of 6 students, ModernBertForTokenClassification)
• crf.safetensors — averaged CRF transition parameters → LinearCRF(num_tags=2, batch_first=True) via safetensors.torch.load_file
• crf.py — vendored, dependency-free LinearCRF (NLL + Viterbi + pack_valid)
• config.json, tokenizer.json, tokenizer_config.json, README.md

Citation

If you use this, please cite the LettuceDetect / s-nlp tool-calling hallucination work, the model-souping recipe (Wortsman et al. 2022), and the ModernBERT base model.