ArcANE-8B-SFT
🏆 Accepted to EMNLP 2026 Main Conference
ArcANE-8B-SFT is a Qwen3-8B model fully fine-tuned on the supervised split of ArcANE (Arc-Aware Narrative Evaluation). It is designed for English role-playing responses that reflect a character's behavioral state at a specified point in a narrative, rather than treating the character as a fixed persona.
Model details
| Field | Value |
|---|---|
| Base model | Qwen/Qwen3-8B |
| Parameters | 8B class |
| Training stage | Supervised fine-tuning (SFT) |
| Parameter update | Full fine-tuning |
| Recommended mode | Qwen3 non-thinking mode |
Intended use
ArcANE-8B-SFT is intended for research on:
- point-in-time character role-play;
- character responses conditioned on a chapter-truncated Character Arc;
- behavioral and value changes across narrative phases;
The strongest evaluated setup supplies the relevant Character Arc only up to the queried chapter. Future phases must not be exposed to the model.
Training data
The SFT split is derived from 12 training novels in the ArcANE corpus, covering 55 characters and 339 character axes. For each (probe, phase) pair under Arc context, three teacher completions were sampled from gpt-5.4-mini. The teacher privately received the phase reference, but each stored training row retained only the character system prompt, scenario-question user prompt, and answer. Reference text was therefore not included in the model input.
The main validated evaluation slice is held out at the novel, character, arc, and probe levels from the training pool.
Training parameters
| Hyperparameter | Value |
|---|---|
| Epochs | 1 |
| Learning rate | 1e-5 |
| Effective batch size | 64 |
| Maximum sequence length | 8,192 tokens |
Reproducibility
The released recipe is training/sft/configs/full/sft.yaml in the ArcANE repository. From training/sft, run bash scripts/train_sft_full.sh.
Evaluation
The paper evaluates free-form role-playing responses on a held-out five-novel slice containing 25 principal characters, 205 arcs, and 1,754 probes. A separate DeepSeek-V4-Flash judge scores four 1 to 100 metrics:
- APF: Action Phase-Fidelity;
- RPF: Reasoning Phase-Fidelity;
- RAE: Reasoning-Action Entailment;
- PTF: Phase Trajectory Fidelity.
Scores are pooled within each novel, novels receive equal weight, and Overall is the mean of the 12 probe-category by metric cells. Higher is better.
Held-out results with Arc context
| Probe category | APF | RPF | RAE | PTF |
|---|---|---|---|---|
| In-Scenario | 57.3 | 55.9 | 49.2 | 51.6 |
| In-World | 56.5 | 56.0 | 48.6 | 45.1 |
| Out-of-World | 56.7 | 55.8 | 48.8 | 46.7 |
| Comparison | Overall |
|---|---|
| ArcANE-8B-SFT, Arc context | 52.3 |
| ArcANE-8B-SFT, strongest non-Arc context | 46.4 |
| Qwen3-8B, Arc context | 43.1 |
The Arc-context score is 5.9 points above this checkpoint's strongest non-Arc context and 9.2 points above Qwen3-8B under the same Arc context. These results measure ArcANE phase fidelity, not general knowledge, safety, or broad instruction-following quality.
Usage
Use the Qwen3 chat template with thinking disabled. The example below is illustrative; replace the compact context with a valid chapter-truncated ArcANE Character Arc.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "holi-lab/ArcANE-8B-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
messages = [
{
"role": "system",
"content": (
'You are <character>, from "<novel>". You are at the point in '
"the story corresponding to chapter <query_chapter>.\n\n"
"Background you have access to:\n"
"<context>\n<chapter-truncated Character Arc JSON>\n</context>"
),
},
{
"role": "user",
"content": "Scenario:\n<scenario>\n\nQuestion:\n<question>",
},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
enable_thinking=False,
return_tensors="pt",
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=1024, do_sample=True, temperature=1.0)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
For faithful point-in-time conditioning, remove all phases later than the query chapter. Always remove literary_validation and evidence_summary; if any later phase is hidden, also remove pole_end and arc_direction. The paper's evaluation drew one sample with backend-default sampling, effectively temperature 1.0, and capped generation at 8,192 tokens.
Citation
@misc{song2026arcaneroleplayinglanguageagents,
title={ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?},
author={Woojung Song and Nalim Kim and Sangjun Song and Chaewon Heo and Jongwon Lim and Yohan Jo},
year={2026},
eprint={2606.05553},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2606.05553},
}