halluscoring-arbert-nli
ARBERT (UBC-NLP/ARBERT) fine-tuned on HalluScoring 2026 Task 1.1 using the NLI framing ([CLS] gold_answer [SEP] model_answer [SEP]). Internally this is run S04 — our best single model of the whole project (clean-dev AUC-ROC 0.9408) and the anchor model for later ensembles, including our best-ever result, S25v2.
Submitted to the competition as SUBMISSION_TEMPLATE_ARBERT_NLI_EDITED.ipynb, alongside the fully-compliant S01 (QA-framed) notebook, with a disclosed edit to the nominally [DO NOT MODIFY] InferenceDataset cell (needed to pass gold_answer through, since the official template's fixed cell only tokenizes question + model_answer). Our submission's own README explicitly deferred to organizer judgment: "if modifying the protected cell is not acceptable, please evaluate the fully compliant notebook." Cross-referencing the official CodaBench leaderboard's NAMAA row against our internal logs indicates the compliant S01 notebook was the one actually scored, not this one — see SYSTEM_WRITEUP.md for the full reasoning.
How to Use
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "HassanB4/halluscoring-arbert-nli"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()
gold_answer = "..."
model_answer = "..."
inputs = tokenizer(gold_answer, model_answer, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits
prob_hallucinated = torch.softmax(logits, dim=-1)[0, 1].item()
print(f"hallucinated={int(prob_hallucinated > 0.5)}, score={prob_hallucinated:.4f}")
Training
| Parameter | Value |
|---|---|
| Base model | UBC-NLP/ARBERT |
| Input format | nli (gold_answer + model_answer) |
| Max sequence length | 512 |
| Batch size | 16 |
| Epochs | 5 |
| Learning rate | 2e-5 |
| Warmup ratio | 0.1 |
| Weight decay | 0.01 |
| Loss | cross-entropy |
| Seed | 42 |
Evaluation
| Split | AUC-ROC | F1-Macro | AUC-PR |
|---|---|---|---|
| Dev (official, n=1300) | 0.9663 | 0.8992 | 0.9790 |
| Dev (clean, unseen-question subset, n=800) | 0.9408 | — | — |
Not independently scored on the hidden test set — the leaderboard-number match described above indicates halluscoring-camelbert-qa (S01) was the notebook actually evaluated, not this one.
Limitations
Requires gold_answer at inference time, which the official submission template's fixed evaluation cell doesn't pass through by default — using this model against that template requires the same disclosed cell edit documented in our submission package.
Citation
@inproceedings{namaa2026halluscoring,
title={{NAMAA at HalluScoring 2026: NLI-Framed BERT Classifiers and Ensembling for Model-Agnostic Arabic Hallucination Detection}},
author={[AUTHOR NAMES TBD]},
year={2026},
booktitle={Proceedings of ArabicNLP 2026},
note={HalluScoring 2026 Shared Task, Track 1}
}