Qwen3-4B Baseline (32k, 6000 steps)
This model is Qwen3-4B trained with standard training (no TTT) on 32k context length for 6000 training steps. This serves as the baseline for comparison with the In-Place TTT version.
🎯 Model Overview
- Base Model: Qwen3-4B (Qwen2.5-4B architecture)
- Training Method: Standard (no Test-Time Training)
- Context Length: 32k tokens
- Training Steps: 6000
- Model Size: 8.3GB
- Format: HuggingFace Transformers (safetensors)
- Training Date: May 13, 2026
📊 Purpose
This is the baseline model for the In-Place TTT research project. It uses the same:
- Base architecture (Qwen3-4B)
- Training data (ProLong 32k)
- Training steps (6000)
- Hyperparameters
The only difference is that it does not use In-Place Test-Time Training.
🔬 Comparison
Compare this baseline with the TTT version to see the impact of In-Place TTT:
| Model | Type | Context | Steps |
|---|---|---|---|
| This Model | Baseline | 32k | 6000 |
| 32k TTT | In-Place TTT | 32k | 6000 |
🚀 Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load model
model = AutoModelForCausalLM.from_pretrained(
"zhongweixie/inplace-ttt-qwen3-4b-32k-baseline",
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
"zhongweixie/inplace-ttt-qwen3-4b-32k-baseline",
trust_remote_code=True
)
# Generate text
prompt = "Your long context prompt here..."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.7,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
📈 Training Details
- Training Data: ProLong pretokenized dataset
- Context Length: 32,768 tokens
- Training Steps: 6,000
- Batch Size: Micro batch size 4, Global batch size 64
- Optimizer: AdamW
- Hardware: 8x H100 GPUs
🔗 Related Models
Part of the In-Place TTT model family:
- Qwen3-4B 32k TTT - TTT version for comparison
- Qwen3-4B 64k TTT - 64k context version
- Qwen3-4B 64k Baseline - 64k baseline
📦 Resources
- GitHub Repository: https://github.com/zhongweixie/inplace_ttt
- Results Dataset: https://huggingface.co/datasets/zhongweixie/inplace-ttt-results
- Paper: https://arxiv.org/abs/2604.06169
📄 License
Apache 2.0 License
🎓 Citation
@inproceedings{feng2026inplace,
title = {In-Place Test-Time Training},
author = {Feng, Guhao and Luo, Shengjie and Hua, Kai and Zhang, Ge and Huang, Wenhao and He, Di and Cai, Tianle},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2026},
note = {Oral Presentation},
url = {https://arxiv.org/abs/2604.06169}
}
📞 Contact
For questions or issues, please open an issue in the GitHub repository.
Model Card Author: Hansirui (zhongweixie)
Uploaded: 2026-09-04
Project: In-Place Test-Time Training