ArcANE-32B-RLVR
🏆 Accepted to EMNLP 2026 Main Conference
ArcANE-32B-RLVR is a Qwen3-32B model trained with SFT, DPO, and reinforcement learning with verifiable rewards. It continues from ArcANE-32B-DPO with Group Relative Policy Optimization (GRPO), using the ArcANE per-phase evaluation rubric as its reward.
Model details
| Field | Value |
|---|---|
| Base model | Qwen/Qwen3-32B |
| Parameters | 32B class |
| Training stage | SFT, DPO, and RLVR with GRPO |
| Parameter update | LoRA, rank 64 and alpha 128 |
| Recommended mode | Qwen3 non-thinking mode |
Intended use
ArcANE-32B-RLVR is intended for research on:
- point-in-time character role-play;
- character responses conditioned on a chapter-truncated Character Arc;
- rubric-based reinforcement learning for narrative phase fidelity;
The strongest evaluated setup supplies the relevant Character Arc only up to the queried chapter. Future phases must not be exposed to the model.
Training data
For RLVR, prompts at each probe's anchor phase are deduplicated by their (system, user) message pair, and the assistant turn is removed so the policy generates fresh rollouts.
The reward is mean(APF, RPF, RAE) / 100, scored by Qwen3.6-27B against the corresponding phase reference. PTF is not part of the reward because each rollout covers one phase.
Training parameters
| Hyperparameter | Value |
|---|---|
| Initialization | ArcANE-32B-DPO |
| Steps / epochs | 2 |
| Learning rate | 1e-5 |
| Train batch size | 64 |
| Rollouts per prompt | 8 |
| Prompt / response length | 8,192 / 2,048 tokens |
Reproducibility
The released recipe is under training/rl in the ArcANE repository.
Evaluation
RLVR is evaluated under Arc context on records shared with the earlier ArcANE-32B stages. A separate DeepSeek-V4-Flash judge scores APF, RPF, RAE, and PTF from 1 to 100. Values in the cross-judge comparison should be compared within a judge column, not across judges.
| Model | Overall |
|---|---|
| Qwen3-32B | 50.0 |
| ArcANE-32B-SFT | 57.3 |
| ArcANE-32B-DPO | 60.2 |
| DeepSeek-V4-Pro | 62.3 |
| ArcANE-32B-RLVR | 68.2 |
These are shared-record results from the RLVR comparison. Under the primary evaluation judge, RLVR improves Overall by 8.0 points over DPO.
Usage
Use the Qwen3 chat template with thinking disabled. The example below is illustrative; replace the compact context with a valid chapter-truncated ArcANE Character Arc.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "holi-lab/ArcANE-32B-RLVR"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
messages = [
{
"role": "system",
"content": (
'You are <character>, from "<novel>". You are at the point in '
"the story corresponding to chapter <query_chapter>.\n\n"
"Background you have access to:\n"
"<context>\n<chapter-truncated Character Arc JSON>\n</context>"
),
},
{
"role": "user",
"content": "Scenario:\n<scenario>\n\nQuestion:\n<question>",
},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
enable_thinking=False,
return_tensors="pt",
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=1024, do_sample=True, temperature=1.0)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
For faithful point-in-time conditioning, remove all phases later than the query chapter.
Citation
@misc{song2026arcaneroleplayinglanguageagents,
title={ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?},
author={Woojung Song and Nalim Kim and Sangjun Song and Chaewon Heo and Jongwon Lim and Yohan Jo},
year={2026},
eprint={2606.05553},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2606.05553},
}