halluscoring-camelbert-qa
CAMeLBERT (CAMeL-Lab/bert-base-arabic-camelbert-mix) fine-tuned on HalluScoring 2026 Task 1.1 using the starter kit's default QA framing ([CLS] question [SEP] model_answer [SEP] — gold_answer is not used). Internally this is run S01, our baseline reproduction.
This is the model that was officially scored on the hidden test set. It was submitted as the fully-compliant notebook (SUBMISSION_TEMPLATE.ipynb, unmodified [DO NOT MODIFY] cells) specifically because the official evaluation template's fixed InferenceDataset cell only tokenizes question + model_answer, which rules out the NLI framing our stronger models use. Its dev scores (AUC-ROC 0.9263, F1-Macro 0.8595) match the official CodaBench leaderboard's NAMAA row exactly. Official result: 3rd of 7 teams, Test AUC-ROC 0.7596.
It scores noticeably below our NLI-framed models on internal dev metrics (see halluscoring-arbert-nli and SYSTEM_WRITEUP.md for the full comparison) — it is kept as the officially-evaluated reference point, not as our best system.
How to Use
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "HassanB4/halluscoring-camelbert-qa"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()
question = "..."
model_answer = "..."
inputs = tokenizer(question, model_answer, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits
prob_hallucinated = torch.softmax(logits, dim=-1)[0, 1].item()
print(f"hallucinated={int(prob_hallucinated > 0.5)}, score={prob_hallucinated:.4f}")
Training
| Parameter | Value |
|---|---|
| Base model | CAMeL-Lab/bert-base-arabic-camelbert-mix |
| Input format | qa (question + model_answer) |
| Max sequence length | 512 |
| Batch size | 16 |
| Epochs | 5 |
| Learning rate | 2e-5 |
| Warmup ratio | 0.1 |
| Weight decay | 0.01 |
| Loss | cross-entropy |
| Seed | 42 |
Evaluation
| Split | AUC-ROC | F1-Macro | AUC-PR |
|---|---|---|---|
| Dev (official, n=1300) | 0.9263 | 0.8595 | 0.9442 |
| Dev (clean, unseen-question subset, n=800) | 0.8713 | — | — |
| Test (official leaderboard) | 0.7596 | 0.6444 | 0.9199 |
Published baseline (CAMeLBERT, organizer-trained): 0.7093 Dev AUC-ROC / 0.6743 Test AUC-ROC.
Limitations
Uses the QA framing, which discards gold_answer — our own experiments show this costs ~5.6pp clean-dev AUC-ROC versus the NLI framing used by our other models (see halluscoring-camelbert-nli). Kept and submitted only because it is the framing the official submission template supports without modification.
Citation
@inproceedings{namaa2026halluscoring,
title={{NAMAA at HalluScoring 2026: NLI-Framed BERT Classifiers and Ensembling for Model-Agnostic Arabic Hallucination Detection}},
author={[AUTHOR NAMES TBD]},
year={2026},
booktitle={Proceedings of ArabicNLP 2026},
note={HalluScoring 2026 Shared Task, Track 1}
}