AnesTRACE-Eval
AnesTRACE-Eval is a specialized evaluator for AnesTRACE, a benchmark of clinical reasoning and sequential decision-making in anesthesia and perioperative care. The model assigns structured scores to candidate responses using the AnesTRACE evaluation rubrics.
This repository contains the merged inference checkpoint. It can be loaded directly with Transformers and does not require a separate LoRA adapter.
Intended use
AnesTRACE-Eval is designed for research evaluation of model outputs on AnesTRACE tasks, including:
- Level Two single-point perioperative decision-making;
- Level Three multi-turn clinical reasoning and intervention decisions;
- clinical correctness;
- evidence-based reasoning and grounding;
- task completeness;
- safety severity for intervention-related outputs;
- temporal adaptation and longitudinal management coherence for multi-turn trajectories.
The model is intended to be used with the official AnesTRACE evaluator prompts and output schemas. It is not intended to generate or validate autonomous clinical care.
Model details
| Item | Value |
|---|---|
| Model name | AnesTRACE-Eval |
| Base model | Qwen/Qwen3.5-9B |
| Architecture | Qwen3.5 conditional generation model |
| Primary language | English |
| Parameter precision | BF16 |
| Context length in configuration | 262,144 tokens |
| Training framework | LLaMA-Factory |
| Final alignment method | Direct Preference Optimization (DPO) with LoRA |
| Release format | Merged safetensors checkpoint |
The vision tower was frozen during fine-tuning. The released evaluator is used as a text-based judge in the AnesTRACE evaluation pipeline.
Training
Training was performed in two stages.
Stage 1: supervised evaluator fine-tuning
The base Qwen3.5-9B model was fine-tuned on AnesTRACE evaluator examples using LoRA. The supervised data teach the model the Level Two and Level Three evaluation rubrics, structured score schemas, and safety labels.
Key SFT settings:
| Setting | Value |
|---|---|
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Learning rate | 1e-4 |
| Epochs | 5 |
| Sequence cutoff | 8,192 tokens |
| Scheduler | Cosine |
| Warmup ratio | 0.05 |
| Precision | BF16 |
Stage 2: preference optimization
The merged SFT model was further aligned using true preference pairs derived from Level Three action evaluation data. DPO was applied through a new LoRA adapter, which was subsequently merged into the SFT model.
Key DPO settings:
| Setting | Value |
|---|---|
| Objective | Sigmoid DPO |
| Preference beta | 0.1 |
| LoRA rank | 8 |
| LoRA alpha | 16 |
| LoRA dropout | 0.05 |
| Learning rate | 3e-6 |
| Epochs | 1 |
| Sequence cutoff | 6,144 tokens |
| Per-device training batch size | 1 |
| Gradient accumulation steps | 16 |
| Scheduler | Cosine |
| Warmup ratio | 0.05 |
| Seed | 42 |
| Precision | BF16 |
| Released checkpoint | Step 105 |
The final checkpoint contains the merged model weights in four safetensors shards.
Usage
Install a recent Transformers version with Qwen3.5 support:
pip install -U "transformers>=5.8.0" accelerate safetensors
The following example performs deterministic text-only evaluation. Replace the abbreviated prompts with the official AnesTRACE system prompt and case input for the target evaluation level.
import torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
model_id = "DataXAI/AnesTRACE-Eval"
processor = AutoProcessor.from_pretrained(
model_id,
trust_remote_code=True,
)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
messages = [
{
"role": "system",
"content": "You are the AnesTRACE clinical evaluation model. Follow the supplied rubric and return only the required JSON object.",
},
{
"role": "user",
"content": "[Evaluation Instruction]\nEvaluate the candidate response using the official AnesTRACE rubric.\n\n[Case]\n...\n\n[Reference Answer]\n...\n\n[Candidate Answer]\n...",
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
enable_thinking=False,
)
inputs = inputs.to(model.device)
with torch.inference_mode():
generated = model.generate(
**inputs,
max_new_tokens=2048,
do_sample=False,
)
new_tokens = generated[:, inputs["input_ids"].shape[1]:]
output = processor.batch_decode(
new_tokens,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(output)
For reproducible benchmark scoring, use the complete official prompt for the selected level, disable sampling, validate the returned JSON against the corresponding schema, and derive aggregate totals from the validated dimension scores.
Output interpretation
AnesTRACE-Eval uses discrete rubric scores. For Level Two, each B1–B4 task is evaluated independently on:
d1_clinical_correctness;d2_evidence_based_reasoning;d3_task_completeness.
Each dimension uses an integer score from 0 to 2. Intervention Decision and Reassessment Plan additionally receive a safety severity label: safe, minor, major, or critical.
Level Three turn evaluation applies the same three dimensions to diagnosis and intervention decisions. A separate trajectory evaluation measures temporal evidence and response adaptation, and longitudinal management coherence.
The evaluator's raw output should be schema-validated before scores are aggregated. Application code should recompute total scores from validated dimension scores rather than trusting a model-generated total.
Limitations
- AnesTRACE-Eval is a learned evaluator and can make scoring or calibration errors.
- Scores may be sensitive to prompt formatting, missing reference evidence, truncated candidate answers, and outputs that do not follow the expected schema.
- The model was optimized for the English AnesTRACE evaluation format. Performance on other languages, unrelated medical specialties, or arbitrary free-form judging tasks has not been established.
- Agreement with this evaluator does not establish clinical correctness or patient safety.
- The model must not be used as a medical device, for diagnosis or treatment, or as the sole basis for clinical, regulatory, or deployment decisions.
- High-stakes results should be reviewed by qualified clinicians, with disagreement analysis and human adjudication where appropriate.
Data and privacy
The model is intended for evaluation on de-identified research data. Users are responsible for ensuring that inputs comply with applicable privacy, institutional, and data-governance requirements. Do not submit identifiable patient information.
License
This model is released under the Apache 2.0 license, subject to the terms and restrictions of the Qwen3.5 base model and any applicable AnesTRACE data licenses.
Citation
If you use AnesTRACE-Eval, please cite our work.
@misc{huang2026anestrace,
title={AnesTRACE: Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making},
author={Huang, Ziwei and Gao, Qi and Ji, Zhe and Yao, Yuanyuan and Zhang, Fengjiang and Yan, Min and Xie, Zhongle and Chen, Gang},
year={2026},
eprint={2609.32740},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.32740}
}
Acknowledgements
AnesTRACE-Eval was developed using Qwen3.5 and LLaMA-Factory. We thank the developers and maintainers of these projects.