Mistral 7B ARC QLoRA with multiple-choice loss
This is the validation-selected checkpoint from an ARC multiple-choice (MC) fine-tuning experiment starting from Mistral-7B-v0.1. It is a task-specific model, not an instruction-tuned chat assistant. The root contains the complete FP16 model and tokenizer.
Recorded evaluation
All three conditions below used the same 1,172 ARC-Challenge test questions, 25-shot context, FP16, batch size 4, SDPA, no chat template and no added BOS/EOS. The two fine-tuned models were evaluated after merging their original saved adapters into a fresh, unquantized FP16 base.
| Model | Normalized correct | acc_norm | Raw correct | acc |
|---|---|---|---|---|
| Base Mistral-7B-v0.1 | 719 / 1,172 | 61.35% | 668 / 1,172 | 57.00% |
| Answer-only SFT reference | 787 / 1,172 | 67.15% | 744 / 1,172 | 63.48% |
| MC loss, selected epoch 1 | 953 / 1,172 | 81.31% | 918 / 1,172 | 78.33% |
MC improves acc_norm by 19.97 percentage points over the base and 14.16 points over the SFT reference.
Training and selection
Training used the same fixed, de-duplicated set of 3,353 ARC-Challenge and ARC-Easy training questions as the SFT reference, retaining all original answer choices. Training questions whose normalized text matched held-out questions were excluded. Held-out answer labels were not used for training. This checks the specified text overlap rule, not all possible semantic overlap or the base model's pretraining contamination.
For each question, the model scores each answer completion after Question: {question}\nAnswer:.
Its score is the sum of answer-token log probabilities divided by Python len(original_choice_text), including original spaces but excluding the added separator space.
Cross-entropy over these character-normalized choice scores (temperature 1) is averaged per question.
Only answer-token log probabilities contribute to each choice score; prompt and padding positions are excluded from the sum.
The softmax over choice scores is a relative distribution, not calibrated confidence.
QLoRA used NF4 with double quantization and BF16 computation; LoRA rank 8, alpha 16, dropout 0.05, applied to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj.
Training stopped early at epoch 11. Epoch 1 / step 210 was selected using the highest ARC-Challenge validation normalized accuracy: 237 / 299 (79.26%), evaluated with NF4/BF16. Ties retain the earlier checkpoint. Selection was frozen before the final test.
Revisions and representation
- Base and tokenizer:
mistralai/Mistral-7B-v0.1, revision27d67f1b5f57dc0953326b2601d68371d40ea8da. - Dataset:
allenai/ai2_arc, revision210d026faf9955653af8916fad021475a3f00453.
The published FP16 weights were produced by merging the selected LoRA adapter into the original, unquantized FP16 base model.
The tokenizer vocabulary is unchanged; saved settings disable added BOS/EOS, use <unk> (ID 0) as PAD and right padding.
Loading the complete model
Install the versions in requirements.txt.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "InkeeJung/mistral-7b-arc-qlora-mc"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(
REPO,
torch_dtype=torch.float16,
device_map={"": 0},
attn_implementation="sdpa",
use_safetensors=True,
).eval()
Limits and license
This is one task-focused run with one training seed. Validation and test scores differ in both data split and numerical representation, so their difference cannot be attributed solely to quantization. Benchmark specialization and pretraining contamination remain possible.
The base was released under Apache 2.0. This derivative retains that license and attributes the base to Mistral AI. The derivative changes consist of the ARC MC-loss fine-tuning described above.