Vietnamese Spell Checker — BARTpho-syllable multi-task (correction + detection)

Mô hình kiểm tra chính tả tiếng Việt: một encoder-decoder BARTpho-syllable với hai đầu ra dùng chung encoder — đầu sinh câu đã sửa (generation) và đầu phát hiện lỗi mức token (detection head). Đánh giá trên VSEC (9.341 câu).

A Vietnamese spelling-error detector/corrector. One BARTpho-syllable encoder-decoder with two heads sharing the encoder: a seq2seq correction head and a token-level error-detection head. Evaluated on the VSEC benchmark (9,341 sentences).

Results on VSEC (official protocol of the VSEC paper)

Detection-head threshold sweep (encoder only, no decoding):

Quick start

import torch, difflib, unicodedata, re
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo = "bravend/bartpho-syllable-vi-spellcheck"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval().cuda()

# --- input normalization (the model was trained on text normalized this way) ---
_TONE = {"òa":"oà","óa":"oá","ỏa":"oả","õa":"oã","ọa":"oạ","òe":"oè","óe":"oé","ỏe":"oẻ",
         "õe":"oẽ","ọe":"oẹ","ùy":"uỳ","úy":"uý","ủy":"uỷ","ũy":"uỹ","ụy":"uỵ"}
_TONE.update({k.capitalize(): v.capitalize() for k, v in list(_TONE.items())})
_TONE_RE = re.compile("|".join(map(re.escape, _TONE)))
def normalize(text):
    text = unicodedata.normalize("NFC", text.strip())
    text = text.translate(str.maketrans({"“":'"',"”":'"',"‘":"'","’":"'","–":"-","—":"-","…":"..."}))
    return _TONE_RE.sub(lambda m: _TONE[m.group()], text)

text = normalize("Tôi đi hoc ở Học viện Công nghệ Bưu Chính Viễn Thông, chuyên nghành công nghệ thông tin.")
enc = tok(text, return_tensors="pt", truncation=True, max_length=1024).to("cuda")

with torch.no_grad():
    # 1) correction
    out = model.generate(**enc, num_beams=1, max_length=1024)
    corrected = tok.decode(out[0], skip_special_tokens=True)
    # 2) token-level error probability from the detection head (encoder only)
    tok_probs = model.detect(enc.input_ids, enc.attention_mask)[0]   # (seq_len,)

print(corrected)   # Tôi đi học ở Học viện Công nghệ Bưu Chính Viễn Thông, chuyên ngành công nghệ thông tin.

Word-level flags with the recommended rule (generation ∪ detection ≥ 0.9)

def word_error_probs(words):
    """P(error) per word = max over the word's sub-word pieces (labels were assigned per word)."""
    ids, spans = [], []
    for w in words:
        p = tok(w, add_special_tokens=False)["input_ids"]
        spans.append((len(ids), len(ids) + len(p)))
        ids.extend(p)
    inp = torch.tensor([tok.build_inputs_with_special_tokens(ids)], device="cuda")
    probs = model.detect(inp)[0]
    off = 1                                  # one leading <s>
    return [probs[off+a:off+b].max().item() if b > a else 0.0 for a, b in spans]

def gen_changed(src_words, pred_words):
    flags = [False] * len(src_words)
    for tag, i1, i2, _, _ in difflib.SequenceMatcher(a=src_words, b=pred_words).get_opcodes():
        if tag in ("replace", "delete"):
            for k in range(i1, i2): flags[k] = True
        elif tag == "insert" and src_words:
            flags[min(i1, len(src_words) - 1)] = True
    return flags

src_words = text.split()
flags = [g or p >= 0.9 for g, p in zip(gen_changed(src_words, corrected.split()),
                                       word_error_probs(src_words))]
print([w for w, f in zip(src_words, flags) if f])   # ['hoc', 'nghành']

Notes:

• trust_remote_code=True is required: the repo ships the multi-task class (modeling_spellcheck.py) and a tokenizer whose SentencePiece model is restricted to the 40K vocabulary (tokenization_spellcheck.py). Loading with the stock classes silently drops the detection head and tokenizes rare strings differently from training.
• Inputs longer than 1024 sub-word pieces must be split (by sentence) before calling the model. The model was trained on single sentences; feed whole paragraphs sentence by sentence.
• Detection-only mode (model.detect) is encoder-only and ~30× faster than generation; use it for triage, and generation for the final decision.
• The detection threshold trades precision for recall: 0.4 gives the best DF; 0.7 gives DP ≈ 0.97 at DR ≈ 0.78 (see the sweep above).

Model description

Training

• Stage 1 — multi-task training (generation loss + detection cross-entropy) on ~12M Vietnamese news sentences with synthetic errors: tone-mark and diacritic confusions, telex/VNI typing errors, keyboard typos, syllable split/merge, phonologically plausible real-word substitutions (word confusion cache filtered to valid syllables), classic compound confusions, and tone-mark misplacement. ~7–10% of samples are kept clean.
• Stage 2 — a 50-step fine-tune mixing 18% self-mined hard negatives (word-level min P(error) ≤ 0.6) with fresh synthetic data, LR 8e-6, effective batch 552. Longer hard-negative training overfit the synthetic distribution and was rejected on held-out data.

Intended use and limitations

• Intended for formal written Vietnamese (news, documents, edited prose).
• Weak on informal social-media text and teencode: on the ViLexNorm test set this checkpoint reaches only F1 ≈ 0.73 for error detection.
• Proper nouns are rarely flagged: the synthetic noise never corrupts mid-sentence TitleCase words, so the model learned that capitalized words are trustworthy.
• Trained on single sentences: multi-paragraph inputs must be sentence-split first, otherwise headline lines and paragraph breaks trigger spurious rewrites.
• Legitimate orthographic variants (i/y as in kì/kỳ, sĩ/sỹ) may be flagged; a small rule layer on top is advisable in production.

License

Released under CC BY-NC 4.0: free to use, share and adapt for non-commercial purposes with attribution. Any commercial or production use requires prior written permission from the author — please contact the author at bravend.dev@gmail.com or through this Hugging Face profile. The base model vinai/bartpho-syllable is MIT-licensed.

Citation

@misc{bravend2026vispellcheck,
  title  = {Vietnamese Spell Checker: BARTpho-syllable multi-task correction + detection},
  author = {Nguyen Duy Dung},
  year   = {2026},
  url    = {https://huggingface.co/bravend/bartpho-syllable-vi-spellcheck}
}

Base model:

@inproceedings{bartpho,
  title     = {{BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese}},
  author    = {Nguyen Luong Tran and Duong Minh Le and Dat Quoc Nguyen},
  booktitle = {Proceedings of INTERSPEECH},
  year      = {2022}
}

Benchmark dataset:

@inproceedings{do2021vsec,
  title     = {{VSEC: Transformer-based Model for Vietnamese Spelling Correction}},
  author    = {Do, Dinh-Truong and Nguyen, Ha Thanh and Bui, Thang Ngoc and Vo, Hieu Dinh},
  booktitle = {PRICAI 2021: Trends in Artificial Intelligence},
  year      = {2021},
  doi       = {10.1007/978-3-030-89363-7_20}
}

Compared system:

@inproceedings{nguyen2023vietnamese,
  title     = {A Vietnamese Spelling Correction System},
  author    = {Nguyen, Thien Hai and Pham, Thinh and Le, Khoi Minh and Luong, Manh and Tran, Nguyen Luong and Man, Hieu and Nguyen, Dang Minh and Luu, Anh Tuan and Nguyen, Thien Huu and Bui, Hung and Phung, Dinh and Nguyen, Dat Quoc},
  booktitle = {Companion Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI '23)},
  year      = {2023},
  doi       = {10.1145/3581754.3584159}
}