Qwen3.5-0.8B Pure Text ZH-EN
A text-only derivative of Qwen/Qwen3.5-0.8B optimized for Chinese and English inference.
Changes
- Converted
Qwen3_5ForConditionalGenerationtoQwen3_5ForCausalLM. - Removed the vision tower and MTP weights.
- Removed visual, grounding, audio, and TTS special tokens.
- Pruned the vocabulary from 248,320 rows to 182,684 real tokens, padded to 182,784 rows for alignment.
- Remapped tokenizer IDs and retained
old2new.jsonfor converting existing tokenized datasets. - Preserved the original weights exactly for every retained vocabulary item; no additional training was performed.
The resulting checkpoint has 685,284,160 parameters and occupies approximately 1.3 GiB in BF16 safetensors format.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "wejyy888/Qwen3.5-0.8B-PureText-ZH-EN-v5.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")
messages = [{"role": "user", "content": "介绍一下杭州"}]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Use a Transformers version that includes Qwen3_5ForCausalLM support.
Compatibility notes
-
Verified environments: transformers 5.2.0 (tokenizers 0.22.2, torch 2.14.0) and 5.17.0. Under 5.2.0 the tokenizer loads at exactly 182,684 tokens with no automatic vocabulary expansion, all 320 tensors load as
Qwen3_5ForCausalLM(685,284,160 parameters), and forward/generation smoke tests pass. -
The linear-attention layers fall back to a pure-torch path when
causal-conv1dandflash-linear-attentionare not installed; install both for faster training and inference. -
Previously tokenized datasets must be remapped with
old2new.json. -
Chinese and English tokenization is preserved exactly after ID remapping.
-
Other languages still decode correctly through byte fallback, but generally use more tokens.
-
This checkpoint accepts text only; image, video, audio, and TTS inputs are unsupported.
-
A short continued-pretraining or SFT stage is recommended before production use to adapt to the reduced vocabulary.
Evaluation
Chinese e-commerce live-search query intent classification: 100 queries, 3 classes (LIVE / BRAND / ITEM), greedy decoding, max 256 new tokens, evaluated with transformers 5.2.0.
| Class | Precision | Recall | F1 |
|---|---|---|---|
| LIVE | 84.2% | 53.3% | 65.3% |
| BRAND | 93.1% | 90.0% | 91.5% |
| ITEM | 73.1% | 95.0% | 82.6% |
Overall accuracy 81.0%, macro-F1 79.8%, and 97/100 per-row agreement with the unpruned source model under the identical prompt. Vocabulary pruning caused no measurable degradation on this task.
Reproduction
prune_qwen35_zh_en.py contains the conversion procedure. The padded vocabulary rows are initialized to zero so they cannot produce invalid high logits.
Validation
- Loaded successfully through
AutoModelForCausalLMasQwen3_5ForCausalLM. - No visual or MTP parameters remain.
- Retained-token logits match the source model exactly in the comparison test.
- Chinese/English tokenizer equivalence, text chat-template encoding, forward pass, and generation smoke tests pass.
- 100-query intent classification end-to-end run under transformers 5.2.0 on Apple MPS reproduces the source model's accuracy (81.0%) with 97% per-row agreement.