ARI-8B

Overview

ARI-8B is a specialized language model for restoring damaged or illegible Hanja characters in Korean historical records, including the Annals of the Joseon Dynasty (AJD) and the Journal of the Royal Secretariat (JRS). It was developed as a computationally efficient alternative to ARI-32B under the same overall training and optimization framework, while retaining strong restoration performance.

It is part of ARI (Archive Restoration Intelligence), a retrieval-augmented historical document restoration framework that combines:

• implicit historical and linguistic knowledge encoded in a pretrained LLM,
• explicitly retrieved historical documents,
• chronological metadata, and
• domain-specific supervised fine-tuning.

This design is especially effective for restoring named entities, such as personal names, locations, organizations, official titles, dates, and book titles, which often cannot be inferred from local context alone.

ARI-8B is the computationally efficient ARI variant, designed to retain strong restoration performance with substantially lower inference cost than ARI-32B.

Model Details

The released model contains the fine-tuned language-model weights only.
To reproduce the full paper setting, users must separately retrieve relevant historical documents and include them in the prompt.

Key Innovations

• RAG-Driven Restoration: Combines the model's implicit knowledge with explicitly retrieved historical records to improve the restoration of proper nouns and other knowledge-intensive spans.
• Named-Entity-Prioritized Training: Applies named-entity-focused masking to 25% of the training data.
• Dynamic Masking: Changes corruption positions across epochs to improve robustness and reduce overfitting.
• Chronological Grounding: Uses metadata such as the reigning king, year, month, and date.
• Retrieval Deduplication: Removes near-duplicate retrieved documents to expose the model to more diverse evidence.

Training Data

ARI was trained using Korean historical Hanja records from AJD and JRS.

After holding out validation, test, and real-world damaged documents, approximately 2.02 million documents were used for training and retrieval.

The training corpus contains more than 16 billion training tokens after prompt construction.

Synthetic Damage Generation

Training and evaluation examples were constructed by replacing original characters with indexed damage markers such as [D1] and [D2].

• Mean corruption rate centered around 2.96%
• At least one damaged character per document
• Corruption spans sampled to reflect real damaged-document statistics
• 75% random masking
• 25% named-entity-prioritized masking
• Dynamic masking across epochs
• Duplicate validation and test examples removed from training

Method

Retrieval

The full ARI pipeline retrieves external evidence using:

• BM25 lexical retrieval
• Top 20 related documents
• retrieval queries built from the damaged document after removing damaged spans
• near-duplicate filtering using a 0.8 string-similarity threshold

BM25 outperformed embedding-based retrieval in the paper. This suggests that lexical character overlap is particularly useful for logographic Hanja text.

Knowledge Integration

The model input combines:

1. task instructions,
2. output-format demonstrations,
3. chronological metadata,
4. the damaged document, and
5. retrieved related documents.

This allows ARI to connect local linguistic context with period-specific historical facts and reduces unsupported restoration guesses.

Fine-Tuning

ARI-8B was trained as a computationally efficient alternative to ARI-32B under the same overall optimization framework.

Performance

The figure compares restoration accuracy across ARI, untuned Qwen3 models, BERT-Res, and general-purpose proprietary LLMs.

• ARI-8B achieves 35.36% on named entities and 76.87% on random characters, substantially outperforming the untuned Qwen3-8B base model.
• ARI-32B achieves the best overall performance with 39.31% on named entities and 80.42% on random characters.

Usage

Input Format

Each damaged character must be replaced by an indexed marker:

傳曰, 同敦寧·[D1][D2]都正竝許遞, 今日政差出.

Each marker corresponds to exactly one original Hanja character.

The expected output is:

{"[D1]": "宗", "[D2]": "親"}

Example with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "DAMI-Lab/ARI-8B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)
model.eval()

system_prompt = '''### Task
You are an expert in restoring damaged Hanja characters. Restore each [Dn] with exactly the original Hanja character. Each [Dn] corresponds to exactly one Hanja character.

### Requirements
Base your restoration on the document’s overall context and meaning rather than treating each damaged token in isolation.

### Input & Output
The input consists of the document itself, its metadata, and the related documents.
While the related documents are omitted in the shot for conciseness, they are always present in the actual dataset.
The output must follow the format: {"[Dn]": "the restored Hanja character for [Dn]"}

### Example Input & Output
[Example 1]
**Input**
The document was written at date: 7, month: 6, year: 1771 - 英祖 era.
Input Document: "傳于[D1]興宗曰, 當自光明殿出, 承旨·侍衛, 來待于建禮門."
**Output**
{"[D1]":"李"}

[Example 2]
**Input**
The document was written at date: 26, month: 8, year: 1824 - 純祖 era.
Input Document: "[D1][D2]口傳政事, 副護軍單李紀淵."
**Output**
{"[D1]":"兵","[D2]":"曹"}

[Example 3]
**Input**
The document was written at date: 30, month: 6, year: 1686 - 肅宗 era.
Input Document: "府[D1]啓, 請[D2]禮·壽進·於[D3]·龍洞·明安公主房折受處[D4]査正事. 入啓."
**Output**
{"[D1]": "前", "[D2]": "明", "[D3]": "義", "[D4]": "一"}

[Example 4]
**Input**
The document was written at date: 14, month: 7, year: 1654 - 孝宗 era.
Input Document: "備[D1]記, 國[D2]難事, 而謀避[D3]免, 朝有逆黨而營救掩護, 最在人先, 身爲[D4][D5], 所爲如此, 他[D6][D7][D8]? 右議政具仁垕罷職."
**Output**
{"[D1]": "忘", "[D2]": "有", "[D3]": "辭", "[D4]": "大", "[D5]": "臣", "[D6]": "何", "[D7]": "足", "[D8]": "觀"}

[Example 5]
**Input**
The document was written at date: 26, month: 4, year: 1656 - 孝宗 era.
Input Document: "又啓曰, 卽者[D1][D2]官, 使差備[D3][D4], 以勅使之意, [D5][D6][D7]臣等, 使之[D8]待於[D9]宴廳, 臣等依其言, 卽[D10][D11]去, 則大通官四人, 一時[D12]來, 傳[D13]禮部咨文[D14]通於臣等曰, 今此咨文, 從速入啓, 明日內回報, 可也[D15][D16], 故咨文送于政院之意, 敢啓. 傳曰, 知道."
**Output**
{"[D1]": "大", "[D2]": "通", "[D3]": "譯", "[D4]": "官", "[D5]": "傳", "[D6]": "言", "[D7]": "於", "[D8]": "來", "[D9]": "西", "[D10]": "爲", "[D11]": "進", "[D12]": "出", "[D13]": "給", "[D14]": "一", "[D15]": "云", "[D16]": "云"}'''

related_documents = [
    "傳曰, 呈告工判·同敦寧許遞, 今日政差出.",
    "傳曰, 在外敦寧都正·同敦寧, 許遞, 今日政差出.",
    "傳曰, 呈告禮曹判書·同敦寧·兵曹參判·同成均許遞, 今日政差出.",
]

related_text = "\n".join(f"- {doc}" for doc in related_documents)

user_prompt = f'''Use the following documents as references to accurately restore the input document.

Related Documents:
{related_text}

The input document was written at date: 28, month: 7, year: 1887 - 高宗 era.

Input Document: 傳曰, 同敦寧·[D1][D2]都正竝許遞, 今日政差出.
'''

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_prompt},
]

try:
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,
    )
except TypeError:
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

generated_ids = output_ids[0, inputs["input_ids"].shape[-1]:]
response = tokenizer.decode(generated_ids, skip_special_tokens=True)
print(response)

For the reported results, use the full ARI prompt and retrieve 20 relevant documents with BM25 before inference.

Dataset Access

The train, validation, and test datasets are available upon request.

Please contact gabeega020@gmail.com for dataset inquiries.

Citation

@inproceedings{kim-kang-2026-leveraging,
  title     = {Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models},
  author    = {Kim, Gabeen and Kang, Kyeongpil},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year      = {2026},
  pages     = {43290--43304},
  doi       = {10.18653/v1/2026.findings-acl.2148}
}

Contributors

• Gabeen Kim — Department of AI Convergence, Kangwon National University
• Kyeongpil Kang — Department of Computer Science and Engineering, Kangwon National University

Acknowledgments

This work was supported by the National Research Foundation of Korea grant funded by the Korean government (MSIT), RS-2025-00555320, and by the National Supercomputing Center with supercomputing resources and technical support under KSC-2024-CRE-0388.