Qwen3-0.6B — RoPE retained, recalibrated on 1B tokens (control model)

A research artifact, not a general-purpose model. This is Qwen3-0.6B recalibrated on 1B tokens of FineWeb-Edu with RoPE left active throughout. It is the matched control for vhallac/qwen3-0.6b-nope-recal-1b, which received identical training — same corpus, same cached token stream, same step count, same schedule, same seed — with RoPE removed.

The only difference between the two models is RoPE. That is the entire point of this checkpoint.

Loading is unremarkable — this is a standard Qwen3 model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "vhallac/qwen3-0.6b-rope-recal-1b"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_id)

(Its sibling does require a runtime patch. Read that model's card before using it.)

Why a control model exists

Part of the rope-as-scaffold research program, which tested whether RoPE is a training scaffold that can be discarded after pretraining.

The problem this checkpoint solves: earlier comparisons pitted a RoPE-removed model that had received 1B extra tokens against an untouched base model that had received none. Any difference confounds "RoPE was removed" with "received extra in-domain training." Since the recalibration corpus is also the evaluation domain, that confound is large and it runs in a direction that flatters the RoPE-removed model.

This model holds the training constant so the comparison isolates RoPE. It changed the program's conclusions: one prior result reversed sign, another was halved in magnitude, and a persistent perplexity penalty became visible that had been invisible against the un-adapted baseline.

Results

Held-out FineWeb-Edu perplexity, fp32, 5M-token frozen eval slice, identical harness for all three:

Reading these correctly matters. Recalibration is worth 0.425 nats (3.0819 → 2.6569). Removing RoPE gives 0.169 nats of that back (2.6569 → 2.8260). The RoPE-removed model's apparent improvement over base (21.80 → 16.88) is the net of those two, and reporting it against base alone makes a real regression look like a gain.

Known limitations

• Effective context ≈ 2048 tokens for the recalibrated behaviour, despite max_position_embeddings: 40960 inherited from base. Unlike its NoPE sibling this model does not collapse beyond 2048 (it inherits base's 32k-context pretraining), but its recalibration only covered 2048.
• English-only recalibration corpus (FineWeb-Edu), while Qwen3 is multilingual + code + math. Expect domain skew relative to base outside the eval domain.
• Not instruction-tuned beyond whatever base carried; recalibration was plain LM training.
• Single seed, single recipe, 0.6B scale.

Training

Trained from the same cached token file as its NoPE sibling, not merely the same corpus — the two models saw the identical token sequence in the identical order. training_metrics.csv and training_manifest.json are included for full provenance.

Note: training_manifest.json in this repo carries "artifact": "qwen3-droped" — a known mislabel from a hardcoded string in the training script. Every functional field is correct for this model, including rotary_patch_applied: false and output_dir: /workspace/qwen3-rope-recal.

Reproduction and full analysis

• Code, specs, and lab notebook: vhallac/crockpot-experiments
• The controlled comparison this model enabled: NOTEBOOK.md, entry 2026-07-28 — RS-amendment-2-3
• Pre-registered plan: RS-amendment-2-3.md; recipe of record RS1-spec.md §11

License

Apache 2.0, inherited from Qwen3-0.6B.