Qwen3-1.7B-BrickGPT (StableText2Brick LoRA, merged)
Qwen3-1.7B fine-tuned on the official BrickGPT / StableText2Brick dataset for text-to-brick (LEGO structure) generation. This is the merged model (LoRA weights merged into the base), ready for inference with transformers/vLLM.
Training details
- Base model: Qwen/Qwen3-1.7B
- Dataset: StableText2Brick (official, 20×20×20 grid) — 47,389 structures × 5 captions, 213,020 train / 23,925 test samples (~291M tokens/epoch)
- Method: LoRA SFT (TRL 0.15 / PEFT 0.15)
- r=16, alpha=32, dropout=0.05, target modules: all linear (q/k/v/o/gate/up/down)
- lr 1e-4, cosine schedule, warmup 100 steps
- effective batch 16 × 8192-token packed sequences (packing), bf16, gradient checkpointing (non-reentrant)
- Hardware: 2× RTX 5090 (DDP), ~14.1 h total
- Epochs: 3 (extended from 2 after eval_loss kept improving)
Results
| epoch | eval_loss |
|---|---|
| 1 | 0.07689 |
| 2 | 0.07369 |
| 3 | 0.07290 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("Kyle308666/BrickGPT-Qwen3-1.7B", torch_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("Kyle308666/BrickGPT-Qwen3-1.7B")
Prompt format follows the Qwen3 chat template; the assistant output is a serialized brick placement sequence (see the BrickGPT paper for the format).
Notes
- Trained for research on structured generation; loss is a weak proxy for this task — evaluate with actual generation + physical stability checks (see BrickGPT paper).
- Base model Apache-2.0 (Qwen3). Dataset: StableText2Brick (see its license terms).