WikiQwen-35B-A3B

Ask anything, get a how-to article. WikiQwen-35B-A3B is Qwen3.6-35B-A3B fine-tuned to answer every message the same way: as a tidy, step-by-step how-to article in markdown, with a picture caption above each step. Hand those captions to WikiQwen-Illustrator-4B and they become illustrations.

Try it first: the WikiQwen Space shows the same article format with WikiQwen-9B, illustrated by WikiQwen-Illustrator-7B.

What It Does

Send it a question ("how do I keep basil alive?"), a problem ("my bike chain keeps falling off") or just "hi", and it sends back a guide. A reply looks like this:

• a # How to … title and a short intro
• one or more ## Method N: or ## Part N: sections, or a single ## Steps
• an [IMAGE: caption] line above each step, written for the Illustrator to draw
• steps written as **N. Bold summary.** Details…, with bullets and > **Tip:** boxes where they help
• optional ## Tips, ## Warnings and ## Things You'll Need at the end

Here is that shape, trimmed. It was written by hand to show the format, so it is not a verbatim model output:

# How to Keep Basil Alive on a Windowsill

Basil wants three things: sun, warmth and a steady drink. Get those right and it will keep you in pesto all summer.

## Part 1: Giving It the Right Spot

[IMAGE: A potted basil plant with bright green leaves sits on a sunny white windowsill, with light streaming through the glass behind it.]
**1. Pick your sunniest window.** Basil needs 6 to 8 hours of direct light a day, so a south-facing window is ideal.
- Give the pot a quarter turn every few days so the plant grows evenly.

[IMAGE: Hands pour water from a small green watering can into the soil of a potted basil plant until it drips into the saucer below.]
**2. Water when the top of the soil feels dry.** Soak the pot until water runs out of the bottom, then empty the saucer.
> **Tip:** Water in the morning so the leaves dry off before the evening chill.

## Warnings
- Don't let the pot sit in standing water. Soggy roots rot quickly.

## Things You'll Need
- A pot with drainage holes
- Potting mix
- A watering can

How to Run WikiQwen-35B-A3B

This is the big one: a mixture-of-experts model with 35B parameters, of which about 3B are active for each token. It writes quickly once loaded, but all 35B parameters have to fit in memory: about 70 GB in bf16. Spread it over several GPUs with device_map="auto", or use one 80 GB card and keep the context modest. The FP8 build needs 37 GB and runs on one GPU with FP8 support.

1. Install the libraries. You need PyTorch, a recent Transformers 5 release with Qwen3.6 support, and Accelerate.

pip install torch "transformers>=5.14" accelerate

• Optional: pip install flash-linear-attention for faster kernels on the linear-attention layers.

2. Load the model and tokenizer. The LoRA is already merged into the weights, so it loads like any Qwen3.6 checkpoint. device_map="auto" shards it across every visible GPU.

3. Ask your question with no system prompt. The article habit is trained in. There was no system prompt in training, and you don't need one now.

4. Turn thinking off. Qwen3.6's chat template can open a <think> block. WikiQwen was trained with it closed, so pass enable_thinking=False. Leave it on and the model muses to itself before it starts writing.

5. Sample like the demo does. Temperature 0.7, top_p 0.9, repetition penalty 1.05 and up to 1,800 new tokens. That is room for about two methods with captions.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "devon7y/WikiQwen-35B-A3B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "How do I keep basil alive on a windowsill?"}]  # no system prompt
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1800, do_sample=True, temperature=0.7, top_p=0.9,
                     repetition_penalty=1.05)
article = tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(article)

6. Pull out the captions if you want pictures. Each [IMAGE: …] line is a ready-made prompt for WikiQwen-Illustrator-4B.

import re
captions = re.findall(r"^\[IMAGE: (.+?)\]\s*$", article, flags=re.M)

Tip: Follow-up messages work (append the reply as an assistant turn and ask again). The model was trained on single questions, though, so each answer is a fresh article, not a chatty reply.

Warnings

• Don't skip enable_thinking=False. It is the one setting that changes everything.
• Long answers can now and then repeat a step. The demo Space stops generating when the same step or caption shows up a third time, and trims an unfinished last line. Do the same if you serve it.

Things You'll Need

• GPUs with about 70 GB of memory between them for the bf16 weights, plus room for the context
• Python with torch, transformers 5.x and accelerate
• A question (any question)

The Family

Training

• Source: about 118k how-to articles from the Kiwix March 2023 archive, with text and Creative Commons illustrations by wikiHow contributors, CC BY-NC-SA 3.0 (see Attribution and License).
• From article to chat: each article became one conversation. The user turn is a realistic message written for that article by Qwen3.6-35B-A3B. The styles range from direct questions to oblique remarks and messages that aren't how-to questions at all, which is how the model learned to answer anything with an article. The assistant turn is the article in the markdown format above, keeping at most its first two methods or parts.
• Captions: every step carries an [IMAGE: …] line. 49% of these captions describe the step's real Creative Commons illustration, written by a vision-language model (Qwen3.6-35B-A3B). The rest were written from the step text alone. The Illustrator was trained on captions in the same style, so the chat model writes prompts the image model already understands.
• Size: 107,054 training conversations and 300 for validation. A 1% bucket of articles never entered training, and the held-out evaluation prompts below come from it.
• Recipe: supervised fine-tuning with LoRA (rank 32, alpha 64, dropout 0.05) targeting the q, k, v, o, gate, up and down projections. In this mixture-of-experts model those names match the 10 full-attention layers (q, k, v, o) and the shared expert in all 40 layers (gate, up, down): 320 LoRA tensors. The routed experts and the linear-attention layers kept their base weights, so the fine-tune is lighter here than on the dense 4B and 9B. Loss was on the assistant turn only. One epoch, learning rate 8e-5 with a cosine schedule and 3% warm-up, effective batch 16, sequences up to 4,096 tokens, bf16, sharded with FSDP across four H100 GPUs. The adapter was then merged into the base weights.

Evaluation

The model wrote one reply per prompt for two prompt sets:

• Held-out prompts: user messages for articles the model never saw in training.
• Everyday prompts: 110 messages that mostly aren't how-to questions (open-assistant questions, everyday small talk and a few handwritten ones like "tell me a joke").

Replies were sampled at temperature 0.7, top_p 0.9 and repetition penalty 1.05, then checked automatically against the article format. These checks measure the shape of the answer. They don't check whether the advice is right.

Limitations

• The advice can be wrong. It sounds confident and looks tidy either way. Be most careful with health, safety, legal and money questions: check anything that matters with a qualified source.
• Almost everything is an article, but less reliably than in the smaller models. Questions and small talk get a how-to article back, so this is not a general assistant. The article habit is looser than in WikiQwen-4B and WikiQwen-9B, whose fine-tune reached more of the network: in the evaluation, 5 of the 110 everyday prompts (writing an email, a haiku, a translation, a rap battle, a role-play turn) got the task done directly, and some articles have explainer titles ("What Does … Mean?") instead of "How to …".
• Knowledge is dated and general. It knows what the base model knows plus a 2023 snapshot of how-to articles. It can't browse, and it may invent product names, measurements or steps.
• Generated articles are not real articles from the source site, and no contributor wrote or reviewed them.
• Pictures can mislead. If you draw the captions with WikiQwen-Illustrator-4B, text inside the pictures comes out garbled, and a picture can show a step wrongly.
• No extra safety training. The fine-tune only taught a format, and the article habit is strong. It may write an article for a request it should turn down, so don't rely on it to refuse.
• English only, and long answers can be cut off at the token limit.

Attribution and License

• Source credit: fine-tuned on article text and Creative Commons illustrations by wikiHow contributors, licensed CC BY-NC-SA 3.0, from the Kiwix March 2023 archive.
• This model and what it writes: CC BY-NC-SA 4.0. Section 4(b) of CC BY-NC-SA 3.0 allows an adaptation to be shared under a later version of the license with the same elements (Attribution, NonCommercial, ShareAlike). In short: credit the source, no commercial use, and share what you make under the same license.
• Base model: Qwen3.6-35B-A3B is released by the Qwen team under Apache-2.0 (license).
• Suggested credit line for articles you share: "Generated by WikiQwen-35B-A3B (CC BY-NC-SA 4.0), a model trained on text and illustrations by wikiHow contributors, CC BY-NC-SA 3.0."
• Independence: WikiQwen is an independent, non-commercial research project. It is not affiliated with or endorsed by the source site or its contributors, or by the Qwen team.

Citation

@misc{wikiqwen_35b_a3b_2026,
  title  = {WikiQwen-35B-A3B: a Qwen3.6-35B-A3B fine-tune that answers as illustrated how-to articles},
  author = {devon7y},
  year   = {2026},
  url    = {https://huggingface.co/devon7y/WikiQwen-35B-A3B}
}