cafebert-ViFactCheck-FC
This model is uitnlp/CafeBERT fine-tuned for VFC-FC on ViFactCheck using the claim paired with full article context.
Evaluation protocol
- Dataset size: 7,232 examples.
- Shared fixed stratified splits for FC and GE: 5,785 train / 723 development / 724 test.
- Labels: Supported, Refuted, and Not Enough Information.
- Fine-tuning seeds: [42, 22, 202].
- Training: 3 epoch(s), AdamW, learning rate 2e-05, weight decay 0.01, warmup ratio 0.1.
- Effective train batch size: 8 (hard-validated against every published run).
- Maximum sequence length: 256.
- Input mode: raw Vietnamese claim and passage.
- The claim is always preserved; only the second sequence (full article context) is truncated when the pair exceeds the encoder limit.
- Topic, author, outlet, URL and other source metadata are excluded from model inputs.
- No class weighting, resampling, retrieval model, sentence ranking, test-time model selection or external evidence is used.
- Checkpoints are selected by development Macro-F1. The representative published checkpoint is seed 202, selected only by development Macro-F1.
Results
Test metrics are reported as mean ± sample standard deviation over seeds [42, 22, 202].
| Metric | Mean ± std |
|---|---|
| Test Macro-F1 | 0.6937 ± 0.0045 |
| Test accuracy | 0.6952 ± 0.0040 |
| Test macro precision | 0.6955 ± 0.0054 |
| Test macro recall | 0.6935 ± 0.0043 |
| Development Macro-F1 | 0.7248 ± 0.0029 |
Per-seed results
| seed | dev_macro_f1 | test_macro_f1 | test_accuracy | micro_batch_size | gradient_accumulation_steps |
|---|---|---|---|---|---|
| 22.000000 | 0.724592 | 0.696041 | 0.697514 | 8.000000 | 1.000000 |
| 42.000000 | 0.722063 | 0.696680 | 0.697514 | 8.000000 | 1.000000 |
| 202.000000 | 0.727870 | 0.688523 | 0.690608 | 8.000000 | 1.000000 |
Label mapping
{
"0": "supported",
"1": "refuted",
"2": "not_enough_information"
}
Usage
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "BaoNhan/cafebert-ViFactCheck-FC"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
claim = "Thông tin này đã được cơ quan chức năng xác nhận."
context = "Bài báo cung cấp bằng chứng liên quan đến phát biểu trên."
inputs = tokenizer(
claim,
context,
return_tensors="pt",
truncation="only_second",
max_length=256,
)
with torch.no_grad():
probabilities = model(**inputs).logits.softmax(dim=-1)[0]
predicted_id = int(probabilities.argmax())
print(model.config.id2label[predicted_id], probabilities.tolist())
Files
aggregate_metrics.json: aggregate metrics and training manifest.artifacts/per_seed_results.csv: one row per fine-tuning seed.artifacts/seed_*_confusion_matrix.csv: confusion matrix for each seed.artifacts/seed_*_classification_report.json: per-class metrics.artifacts/seed_*_test_predictions.csv: IDs, gold/predicted labels and probabilities; raw claims and passages are excluded.
Limitations
ViFactCheck supplies the correct source article and therefore does not evaluate open-web evidence retrieval. VFC-FC can truncate relevant information in long articles and jointly measures verification plus robustness to irrelevant context. VFC-GE uses oracle gold evidence and must not be presented as a realistic end-to-end deployment setting. This model is a research classifier, not an automated arbiter of truth, and may produce confidently incorrect predictions.
Dataset citation
@inproceedings{hoa2025vifactcheck,
title={ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese},
author={Hoa, Tran Thai and Duy, Tran Quang and Tran, Khanh Quoc and Nguyen, Kiet Van},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={39},
number={1},
pages={308--316},
year={2025},
doi={10.1609/aaai.v39i1.32008}
}