DesignForge Qwen2.5-VL 3B (web screenshot -> HTML)
Fine-tune of Qwen/Qwen2.5-VL-3B-Instruct that turns a webpage screenshot + short design brief
into a complete self-contained HTML/CSS/JS document.
Usage
import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"ivan123-123/designforge-qwen2.5-vl-3b", torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained("ivan123-123/designforge-qwen2.5-vl-3b")
msgs = [{"role": "system", "content": "You are DesignForge, an expert front-end engineer and UI designer. Given a webpage screenshot and a design brief, you write one complete, polished, responsive, self-contained HTML document (HTML, CSS and JavaScript as needed). Output only the HTML."},
{"role": "user", "content": [{"type": "image", "image": image},
{"type": "text", "text": "Design brief:
" + brief + "
Generate the complete webpage."}]}]
prompt = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=imgs, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=2500, do_sample=True, temperature=0.6, top_p=0.9, top_k=50)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
Notes
- Trained 200 steps (grad-accum 4, LR 2e-4, cosine) on 504 hand-written WebSight-style brief/screenshot pairs; 500/4 holdout.
- Decode with sampling (temperature ~0.6). The greedy/fp16 path can degenerate on some inputs.
- Processor ships with the training pixel budget
min_pixels=100352, max_pixels=301056. - Held-out quality: 3/3 structurally valid + rendered HTML, mean SSIM 0.67 vs reference screenshots.
- Adapter-only version:
ivan123-123/designforge-qwen2.5-vl-3b-lora(load over the base model withpeft).