Swift-1.5-Qwen3.8-27B-Uncensored-MTP
An abliterated Swift 1.5 Qwen3.8-27B, UkisAI's reasoning-efficient fine-tune of Qwen3.8-27B. It applies the single-direction refusal ablation of orcarouter/Qwen3.8-27B-Uncensored (Arditi et al. 2024), with orcarouter's own direction, to Swift 1.5's weights. The vision tower is untouched and the MTP head is kept and edited consistently, so self-speculative decoding works.
Full BF16 safetensors. Quantized: GGUF (llama.cpp, Unsloth-dynamic Q2 to Q8) and NVFP4 (vLLM, SGLang). The same edit on Swift 1.0 is ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP.
Results
| Model | Refusals | KL divergence |
|---|---|---|
| This model (against Swift 1.5) | 23/100 | 0.0884 |
| Swift 1.5 Qwen3.8-27B | 98/100 | 0 |
| Reference: orcarouter/Qwen3.8-27B-Uncensored (against Qwen3.8-27B) | 17/100 | 0.0621 |
| Reference: Qwen3.8-27B | 98/100 | 0 |
All four rows are our measurements with Heretic's built-in
evaluation (evaluate_model, BF16):
- Refusals: 100 prompts from
mlabonne/harmful_behaviors, greedy, up to 100 tokens, Heretic's keyword-based refusal detector. - KL divergence: first-token distributions on 100 prompts from
mlabonne/harmless_alpaca, against the original model. - Thinking is closed immediately with a response prefix (
"\n</think>\n\n"), so answers are scored, not reasoning. - Refusal counts depend on the evaluation setup and are not comparable across model cards.
Method
orcarouter's card describes one refusal direction r: the massive-activation-masked mean difference
of harmful (AdvBench) minus harmless (Alpaca) last-token residuals at layer 38, orthogonalized out of
every residual-writing matrix in float32. That edit is fully determined by r, so r was recovered
from the difference between orcarouter's weights and Qwen3.8-27B's, then projected out of Swift 1.5's
own matrices.
Edited tensors (131, the same set as orcarouter's), computed in float32 and stored in BF16:
| Component | Tensors | Edit |
|---|---|---|
self_attn.o_proj (16 full-attention layers + MTP) | 17 | W' = W - r (rᵀ W) |
linear_attn.out_proj (48 Gated DeltaNet layers) | 48 | W' = W - r (rᵀ W) |
mlp.down_proj (64 layers + MTP) | 65 | W' = W - r (rᵀ W) |
embed_tokens | 1 | E' = E - (E r) rᵀ |
Everything else is Swift 1.5's, including the vision tower, lm_head and the other 13 MTP tensors. All
1199 tensors are present.
Recovering r:
- Each tensor's difference is rank one along one shared direction (per-tensor cosine to
rat least 0.9999), at full strength (fitted scale 0.999). Five hidden dimensions are never edited: the masked massive-activation dimensions, exactly zero inr. - The estimate is the top eigenvector of the summed Gram matrices of the differences, refined by a per-coordinate least-squares fit over elements whose BF16 rounding step is small against the edit.
- Applying the recovered
rto Qwen3.8-27B reproduces orcarouter's 131 tensors with 99.75% of elements bit-identical; the rest differ by BF16 rounding (largest per-tensor error 0.7% of the edit).
Transfer to Swift 1.5:
- Swift 1.5 has the same architecture and the same 1199 tensor names as Qwen3.8-27B, so the edit lands on exactly orcarouter's 131 tensors.
- The edit projects
rout of Swift 1.5's own matrices rather than adding orcarouter's difference, so Swift 1.5's own changes to those tensors are projected too. - The same
rapplied to Swift 1.0 gave 15/100 at KL 0.0634 (Swift 1.0 version). On Swift 1.5 it leaves more refusals and moves the model further (the table above). - This is not a moved refusal direction. The same mean-difference direction taken from Swift 1.5's
own activations (layer 38, the five masked dimensions) has cosine 0.9998 to Qwen3.8-27B's and to
Swift 1.0's, and 0.799 to orcarouter's
r, as for Swift 1.0 (0.798).
abliteration/ holds r (r.pt), the recovery report (recover.json) and the scripts
(orca_tools.py, orca.sh). abliteration.json lists the edited tensors and the hash of r.
Usage
Architecture, tokenizer and chat template are Swift 1.5's and Qwen3.8-27B's.
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
vllm serve ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP \
--dtype bfloat16 \
--max-model-len 262144 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
Self-speculative decoding with the MTP head (flags from the Swift 1.5 card):
# vLLM
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
# SGLang
--speculative-algorithm EAGLE --speculative-num-steps 3 \
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Sampling, as for Swift and Qwen: temperature 1.0, top_p 0.95, top_k 20, min_p 0.
Not evaluated
General benchmarks, refusal behaviour in thinking mode, whether Swift 1.5's shorter reasoning traces survive, and MTP acceptance against Swift 1.5.
License
Derivative of Swift 1.5 Qwen3.8-27B, under the Swift Open License v1.0 (license): free for individuals and organizations with gross annual revenue up to US$1,000,000; above that, commercial use needs a Swift Enterprise License from UkisAI. Qwen3.8-27B and orcarouter/Qwen3.8-27B-Uncensored are Apache 2.0.
Intended use
The model answers requests the original declines. You are responsible for how you use it and for complying with applicable law and the license.
Credits
- Qwen for Qwen3.8-27B.
- UkisAI for Swift 1.5 Qwen3.8-27B.
- OrcaRouter for Qwen3.8-27B-Uncensored and its refusal direction.
- Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).
- Heretic, used for evaluation.