Qwen3-4B Devin SFT (Instruct)
Fine-tuned Qwen3-4B on Devin session data with tool-calling format. Trained on a GB10 (NVIDIA DGX Spark) with 128GB VRAM.
Training Details
- Base model: Qwen/Qwen3-4B
- Dataset: davidnichols-ops/adaptive-operator-v4-dataset (4,992 samples)
- Format: Chat messages (system/user/assistant) with tool calls in
<tool>{json}</tool>format - Epochs: 3
- Batch size: 16 (effective 32 with grad accumulation)
- Learning rate: 2e-5, cosine schedule
- Precision: BF16
- Hardware: NVIDIA GB10 (128GB VRAM, CC 12.1)
- Training time: ~48 min (cleaned instruct data, no control tokens)
Tool Calling Format
The model expects tool calls in this format:
<tool>
{"name": "shell", "arguments": {"command": "uv run pytest"}}
</tool>
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"davidnichols-ops/qwen3-4b-devin-sft",
dtype=torch.bfloat16,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("davidnichols-ops/qwen3-4b-devin-sft")
messages = [
{"role": "system", "content": "You are a helpful engineering assistant."},
{"role": "user", "content": "Run the test suite and fix any failures"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Speculative Decoding
Pair with the draft model davidnichols-ops/qwen3-0.6b-devin-draft for 2-3x inference speedup. The 0.6B draft model proposes tokens; this 4B target model verifies them.