Qwen3.5-0.8B - Reddit conversational LoRA (test artifact)
This repository is named by its training sources: the suffixes askhis asksci cmv depth eli5 neutral philo sci askred unpop stand for AskHistorians, askscience, changemyview, DepthHub,
explainlikeimfive, NeutralPolitics, philosophy, science, TrueAskReddit and unpopularopinion. Exact
row counts per source are in training/sources.json.
This is not a real fine-tune. It exists to test one thing: whether a small hybrid-attention model can be trained end to end in an awkward setup. Do not use it for anything factual, do not judge model quality by it, and do not cite it as a capability result. The numbers below describe the pipeline, not a usable model.
A LoRA fine-tune of Qwen/Qwen3.5-0.8B, trained on
question/answer pairs from EnDevSols/reddit-qa-dataset. The goal was to see how far an older,
memory-limited NVIDIA GPU can go in 2026, so everything about the recipe is constrained: fp16 only,
no Triton kernels, no 4-bit.
Two variants live in this repository:
| Path | What |
|---|---|
/ (root) | merged fp16 model, Qwen3_5ForCausalLM, vision tower removed, ready to run |
adapter/ | the LoRA weights (39 MiB) to apply on top of the base |
training/ | hyperparameters and the loss/eval records |
What was trained on
- 6,000 train / 256 eval rows, drawn from 10 subreddits, balanced to roughly 600 per source.
- Each row is a Reddit post title (plus selftext when the post had one) mapped to its top-voted answer comment.
- Filters: answer score >= 8, answer length 80-2600 characters, no deleted or removed content, no rows over 480 tokens after tokenisation, exact deduplication.
- Corpus spans 2011 to 2024, so the register is dated in places.
Method
| Item | Value |
|---|---|
| Method | LoRA, fp16 weights, fp32 adapters, autocast with a small fixed loss scale |
| Rank / alpha | r=16, alpha=32, dropout 0 |
| Target modules | 10 Linear names across Gated DeltaNet, Gated Attention and FFN |
| Trainable params | 10.22 M of 752.4 M (1.4 %) |
| Sequence length | 512 max, batch 1 per micro-step, gradient accumulation 16 |
| Optimiser | AdamW, lr 2e-4, cosine decay, 3 % warmup, grad clip 1.0 |
| Epochs | 1 (375 steps, 653,385 supervised tokens) |
| Loss masking | assistant tokens only |
| Memory tricks | gradient checkpointing, plus a chunked cross-entropy over the 248,320-token vocabulary |
| Hardware | one older GPU, fp16 only |
| Software | torch 2.14.1+cu126, transformers 5.19.0, peft 0.21.2, Python 3.12 |
| Wall time | 4 h 11 min |
Cross-entropy is computed in 64-token slices, because full fp32 logits over a 248k vocabulary cost close to 500 MiB per 512-token sequence, and that is what actually limits this setup.
Results
Held-out perplexity on 200 evaluation rows (21,900 supervised tokens):
| Model | Eval loss | Perplexity |
|---|---|---|
| Base (vision-stripped, fp16) | 3.2761 | 26.47 |
| This fine-tune | 2.9616 | 19.33 |
So a 22 % perplexity reduction on Reddit-style question/answer text. Treat it as a plumbing result: it shows the recipe runs and moves the loss on hardware with fp16-only and no flash-attention, where the linear-attention layers fall back to reference PyTorch kernels.
Behaviour change, stated plainly
The base model is post-trained to produce long, structured, markdown-heavy answers. After one epoch of Reddit answers it gets much shorter and much more casual. On greedy/sampled prompts of the form "explain X", mean generation length dropped from the 200-token cap to about 30 tokens.
You gain brevity and tone; you lose depth, and sometimes accuracy. On a question like "How did medieval people keep food from spoiling?" the base model gives a long, mostly correct breakdown while this model answers in two sentences and states something false about refrigeration. Use it for tone and brevity, not for factual depth.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
# substitute your own namespace, or "Qwen" style org if mirrored
repo = "<your-username>/Qwen3.5-0.8B-askhis-asksci-cmv-depth-eli5-neutral-philo-sci-askred-unpop"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="float16", device_map="auto")
msgs = [{"role": "user", "content": "What is a non-obvious sign that a coworker is burned out?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False,
return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
To apply the adapter to the base instead:
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype="float16")
model = PeftModel.from_pretrained(base, "<your-username>/Qwen3.5-0.8B-askhis-asksci-cmv-depth-eli5-neutral-philo-sci-askred-unpop",
subfolder="adapter")
Note the adapter targets the text-only architecture Qwen3_5ForCausalLM with the vision tower
removed. Applying it to the full multimodal checkpoint may need the visual weights dropped first.
Intended use and limits
- Research and prototyping on conversational style transfer for small hybrid-attention models.
- Not a capability upgrade: knowledge, reasoning and factuality are whatever Qwen3.5-0.8B already is.
- One epoch, so the adapter is undertrained by normal standards; more epochs on this mix would probably shorten and coarsen the answers further.
- Reddit text carries slang, dated references and occasional rudeness. Nothing was filtered for toxicity beyond removing deleted content and non-text dumps.
- Trained on English data only, although the base model supports 201 languages.
Licence
Apache-2.0, inherited from the base model. The training data is also Apache-2.0.