Qwen3-1.7B Frontend SFT

A QLoRA supervised fine-tune of Qwen/Qwen3-1.7B for front-end code generation: complete HTML documents, CSS, JavaScript, and whole-file edits.

The repository root holds the merged 16-bit model (loads directly, no PEFT needed). The LoRA adapter alone is under adapter/.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "panzarasa/qwen3-1.7b-frontend-sft"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "You are an expert front-end engineer. Given a website description, reply with a single complete HTML document with its CSS in a <style> block."},
    {"role": "user", "content": "A landing page for a coffee roastery: hero, three product cards, newsletter form, footer."},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=8192, temperature=0.7, top_p=0.9, repetition_penalty=1.05)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

Sampling settings

temperature=0.7, top_p=0.9, repetition_penalty=1.05, max_new_tokens=8192

These are a reasonable default, not a fix for the failure mode below. Both 0.3 and 0.7 were measured on the same six held-out pages and scored the same; the rows that fail differ, the count does not.

Known failure mode: long pages may not terminate

On long full-page generation (the webcode2m style: a real-world page with a trailing <style> block), the model can fall into a repetition loop and burn the whole token budget without emitting </html>. Two shapes were observed:

• textual -- a sentence or a counter repeats (vitamin B580, B581, ...);
• structural -- markup cycles (one generated page held 518 <li> elements drawn from 8 distinct variants).

Measured on 6 random held-out webcode2m rows, at both temperatures: 4 of 6 reached </html>; 2 of 6 hit the 8192-token cap. Raising the temperature does not reliably help. Detect it rather than prevent it: check that the output ends in </html> and retry if it does not.

Short outputs -- single components, JavaScript functions, whole-file edits, the websight style of compact Tailwind page -- did not show this in testing.

Prompt format

The model was trained with the Qwen/Qwen3-4B-Instruct-2507 chat template, not the hybrid-thinking template that ships with Qwen3-1.7B. The tokenizer in this repo already carries the right one. There are no <think> blocks in the output.

System prompts seen during training:

• You are an expert front-end engineer. Given a website description, reply with a single complete HTML document ...
• You are an expert front-end engineer. Given a page brief, reply with a single complete HTML document with its CSS in a <style> block.
• You are a senior front-end engineer. Answer with correct, production-ready HTML, CSS and JavaScript. Be concise.
• You are an expert front-end engineer. Apply the requested change to the file and output only the complete updated file. No explanation, no markdown fences.

Training

Rows longer than 8,192 tokens were dropped, not truncated, so the model never saw an unterminated document.

Data

267,202 training rows / 1,981 validation rows, mixed from:

Each source keeps its own licence; check them before redistributing derived data.

Evaluation

Eval loss decreased monotonically over the single epoch and flattened at the end:

Held-out generation

One held-out row per source was generated and read by hand:

Six further random webcode2m rows were generated at two temperatures and scored on closure, <style> presence, tag balance and repeated blocks. Model: 2/6 at either temperature. The reference pages score 3/6 on the same metric -- real scraped pages are themselves repetitive, so the gap is narrower than the raw number suggests.

Training data duplication

xcodemind/webcode2m is a web scrape, and the same site template recurs many times. Hashing each page's tag sequence (structure only, ignoring text and attributes) across the 59,566 training rows:

• 40,373 distinct structures; 32.2% of the rows are structural duplicates
• 419 structures appear 10+ times; the largest cluster holds 171 copies

There is no exact train/val leakage (0 identical documents), but 52 of the 390 distinct validation page titles also occur in training. The effect is visible in generation: on two of the six sampled rows the model reproduces the held-out reference at 99.6% and 98.1% similarity, including a 2,091-char verbatim span. Temperature does not suppress it.

Practical consequence: eval loss understates the true generalization gap on long pages, and output for this kind of page may closely resemble a real website. Deduplicating by structural hash before the split would remove ~19,000 rows (~20% of all training tokens) and is the first change worth making.

Limitations

• Long full-page generation fails to terminate roughly 1 time in 3; check for </html> and retry.
• Can reproduce training pages near-verbatim on the webcode2m style -- see Training data duplication. Do not assume a generated page is original.
• Can skip elements a brief explicitly asks for (a <form> was missed in one held-out page that requested one).
• Trained for one epoch; eval loss had flattened under a cosine schedule decayed to 1e-6. More epochs on this data would deepen the duplication problem, not fix it -- deduplicate first.
• English only.
• 1.7B parameters: expect scaffolding and structure, not production copy.