MindCube latent-reasoning checkpoints
LoRA adapters for Qwen/Qwen2.5-VL / Qwen2.5-VL-3B-Instruct that reason over
MindCube spatial questions in a compressed latent
vocabulary — the model emits <latent_start> <lat_…> <latent_end> <answer>X</answer> — instead
of a natural-language chain of thought. Each <lat_i> is one BPE-merged unit standing for several
base tokens, so the reasoning decodes in far fewer steps at no accuracy cost.
Training traces (text): leapeto/mindcube-latent-data.
Images: Inevitablevalor/MindCube data.zip.
Checkpoints
Frozen-3B reference ≈ 37.8%; uncompressed full-CoT SFT baseline (plain_3b_ep5) = 46.6%. Accuracy
is flat at ~49–51% across the whole 1.5×–4.05× compression sweep — compression is free down to
the merge-table ceiling. Tinybench = MindCube's 1,050-question eval split, greedy two-stage decode.
| adapter | base | target compression | M (<lat_i>) | median decode tok | tinybench acc |
|---|---|---|---|---|---|
plain_3b_ep5 | 3B | 1.0× (uncompressed full-CoT) | — | 321 | 46.6% |
c1.5mn_3b_ep5 | 3B | 1.5× | 1,961 | 207 | 49.7% |
c2.0mn_3b_ep5 | 3B | 2.0× | 1,983 | 145 | 51.1% |
c2.5mn_3b_ep5 | 3B | 2.5× | 2,082 | 124 | 51.0% |
c3.0mn_3b_ep5 | 3B | 3.0× | 2,294 | 102 | 47.6% |
c3.5mn_3b_ep5 | 3B | 3.5× | 3,011 | 88 | 50.1% |
c2.0mn_7b_ep4 | 7B | 2.0× | 1,983 | 144 | 49.6% |
ffrsn_3b_ep5 | 3B | 2.0× (free-form reasoning target, no map) | — | 33 | 51.9% |
c4.0mn2_3b_ep5 | 3B | 4.0× | 9,081 | 76 | 49.3% |
c4.5mn2_3b_ep5 | 3B | 4.05× (merge ceiling) | 11,895 | 73 | 49.1% |
c2.0mn2_3b_ep5 | 3B | 2.0× (regen-corpus anchor) | 1,993 | 150 | 49.1% |
The mn2 checkpoints are the high-compression extension: same recipe, trained on a regenerated
(self-distilled) copy of the trace corpus. c2.0mn2 reproduces the original c2.0mn within noise
(49.1% vs 51.1%) and exists only to confirm the corpora are comparable. Targets 4.5× and 5.0×
both collapse to the same 4.05× vocabulary — the merge table runs out of recurring structure — so
c4.5mn2 is the ceiling.
Usage
Each adapter dir bundles its tokenizer (defines the <lat_i> tokens) and — for the mn2
checkpoints — a latent_vocab.json sidecar mapping every <lat_i> back to its base-token span
(needed only to expand the latent output into readable text or to retrain; plain inference does
not need it). Decode in two greedy stages with a logit mask:
- Latent — from the prompt +
<latent_start>, restrict next-token logits to the<lat_i>units ∪<latent_end>; stop on<latent_end>(or a cap ~400). - Answer — block the
<lat_i>units and<latent_end>; decode<answer>X</answer>.
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel
import torch
name = "c4.0mn2_3b_ep5"
proc = AutoProcessor.from_pretrained(f"leapeto/mindcube-latent-ckpts/{name}")
base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2.5-VL-3B-Instruct", dtype=torch.bfloat16, device_map="cuda")
base.resize_token_embeddings(len(proc.tokenizer)) # absorb the added <lat_i> units
model = PeftModel.from_pretrained(base, f"leapeto/mindcube-latent-ckpts/{name}").eval()
# then the two-stage masked greedy decode above.
M-RoPE gotcha: in a manual KV-cache decode loop, pass
cache_positionexplicitly every step. Qwen2.5-VL derives decode positions fromcache_position + rope_deltas; omitting it feeds every generated token at position 0 and produces repetition/garbage (transformers.generate/ vLLM handle this automatically).
Prompt format: MindCube aug_cgmap_ffr_out. Benchmark: MindCube (arXiv:2506.21458).
Full-parameter checkpoints — the ground-truth-corpus line (docs 45–54)
Everything below is full fine-tuning (ViT included) of Qwen/Qwen2.5-VL-3B-Instruct on
MindCube's ground-truth map+reasoning corpus (not LoRA, not self-distilled). Each directory is a
complete HF model (4 fp32 safetensors shards, ~16 GB, + processor/tokenizer): load it directly with
Qwen2_5_VLForConditionalGeneration.from_pretrained(dir) — no PEFT, no resize_token_embeddings.
Eval protocol for every number: greedy, N=1,050 tinybench, best of ep4–6.
| dir | what | tinybench acc |
|---|---|---|
plain_s42_ep3/ | stage-1 base: 3 epochs plain SFT on the GT corpus, seed 42. Warm-start base of every arm below and of the latent arms of docs 46–53. | 65.9 (doc 45) |
text_baselines_s777/answer_only_ep{4,5,6}/ | + 6 epochs, target = <answer> only (no reasoning). Rerun pass; first pass scored 70.19. | 69.62 (ep6) |
text_baselines_s777/concise_ep{4,5,6}/ | + 6 epochs, compact map + boilerplate-free sentences (1.91× fewer decode steps) | 74.38 (ep5) |
text_baselines_s777/cod_ep{4,5,6}/ | + 6 epochs, Chain-of-Draft targets (2.29×) | 74.10 (ep5) |
text_baselines_s777/tokenskip2.4_ep{4,5,6}/ | + 6 epochs, LLMLingua-2-pruned targets (2.32×) | 66.76 (ep5) |
Reference points at the same recipe and seed (checkpoints pruned, decodes in the data repo):
plain +6 epochs = 70.95; latent BPE vocab cb 1.8× / 2.0× / 2.4× = 67.81 / 64.48 / 67.33.
Report: docs/54_TEXT_COMPRESSION_BASELINES.md in the code repo (branch mindcube); training data
under gt_corpus/text_baselines/ in leapeto/mindcube-latent-data. Recipe: warm-start from
plain_s42_ep3, lr 1e-5 cosine, effective batch 512, bf16 FSDP, seed 777; on 32 GB cards use
micro-batch 1 × grad-accum 128.
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from huggingface_hub import snapshot_download
import torch
d = snapshot_download("leapeto/mindcube-latent-ckpts", allow_patterns=["text_baselines_s777/cod_ep5/*"]) + "/text_baselines_s777/cod_ep5"
proc = AutoProcessor.from_pretrained(d)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(d, dtype=torch.bfloat16, device_map="cuda").eval()
# plain greedy generate on the MindCube `plain_cgmap_ffr_out` prompt; parse the letter from <answer>…</answer>