AStar-Thought-Latent-Qwen3.5-0.8B-deepseek-v3.2-speciale-openr1-math-3k-filtered-v10-angle90-Qwen3.5-9B-v2.12-strc0.3
This model is a fine-tuned version of Qwen/Qwen3.5-9B on the AStar-Thought-Latent-Qwen3.5-0.8B-deepseek-v3.2-speciale-openr1-math-3k-filtered-v10-angle90 dataset.
Model description
This is the A*-Thought-V2 model introduced in the paper A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM. A*-Thought-V2 is an explicit–implicit interleaved latent reasoning architecture guided by LLM dynamics. It projects question, step, and solution representations into a 3D PCA space to measure the alignment of each reasoning step with the global question-to-solution direction: aligned steps remain explicit text, while deviating steps are compressed into continuous latent tokens. This enables lossless compression of chain-of-thought while improving efficiency.
The paper reports improvements of up to 2.6% in average accuracy across six in-domain and out-of-domain benchmarks on Qwen3.5-9B and Qwen3.6-27B, with up to 2× reduction in response length.
The code for compression, training, and evaluation is available at the A*-Thought GitHub repository. Note that A*-Thought-V2 requires a modified vllm build for efficient inference, as described in the repository.
Intended uses & limitations
This model is intended for research on efficient reasoning in large language models. It uses a custom latent-token architecture, so users should refer to the accompanying codebase and use the modified vllm for inference, including the max_latent_count and max_latent_len hyperparameters described there.
Training and evaluation data
The model is trained on a compressed version of reasoning traces derived from the OpenR1-Math-3k dataset (DeepSeek-V3.2-Speciale), obtained via the A*-Thought-V2 compression pipeline with a PCA angle threshold of 90 degrees. See the GitHub repository for details.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 8
- total_train_batch_size: 64
- total_eval_batch_size: 64
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 0.1
- num_epochs: 3.0
Training results
| Metric | Value |
|---|---|
| train_loss | 0.3914 |
| train_runtime | 13643.8353 s |
Framework versions
- Transformers 5.2.0
- Pytorch 2.10.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2