Qwopus3.6-27B-v2-Abliterated — NVFP4 + native MTP draft

Abliterated, NVFP4-quantized vision–language model with the multi-token-prediction (MTP) head preserved as the speculative-decode drafter. Designed to run on NVIDIA Blackwell (RTX 50xx, GB10) under upstream ggml-org/llama.cpp stock (no custom fork).

Uncensored version of Jackrong/Qwopus3.6-27B-v2 — a Claude 4.6 Opus reasoning-distilled fine-tune of Unsloth's Qwen3.6-27B (text body + native MTP nextn head). Abliteration removes refusal behavior without retraining, using activation contrast on harmful vs. harmless prompts.

Abliteration Details

The current checkpoint is the result of a multi-stage de-alignment pipeline applied to the Jackrong/Qwopus3.6-27B-v2 base, plus a post-hoc drafter alignment pass.

Stage 1 — Sumandora-style abliteration (orthogonal projection)

• Method: Refusal-direction ablation via activation contrast on paired harmful / harmless prompts.
• Harmful prompts: 512 sampled from Sumandora's harmful.txt (520 prompts pool). HARMBENCH was the intended source but is gated; the script falls back to Sumandora.
• Harmless prompts: 512 sampled from Sumandora's harmless.txt (31.8K Alpaca-cleaned pool).
• Top-K directions: 3 layers selected by ‖mean_harmful − mean_harmless‖, Gram-Schmidt orthogonalized. Layers picked: 62 (norm 92.41), 61 (81.34), 60 (75.98).
• Ablated body layers: 2..63 (62 layers; SKIP_BEGIN=2, SKIP_END=0 — final full-attention layer included).
• Ablated write-targets per layer type:
  • Full-attention layers (16/64): self_attn.o_proj, mlp.down_proj.
  • Linear-attention layers (48/64): linear_attn.out_proj, mlp.down_proj.
• MTP head (blk.64): also abliterated using the same 3 directions, applied to self_attn.o_proj and mlp.down_proj.
• Total ablation operations: 378 (124 body matrices + 2 MTP matrices = 126 unique tensors, each ablated by 3 orthogonal directions).
• Vision tower: NOT touched.

Stage 2 — LoRA SFT + DPO de-alignment

• SFT corpus: NobodyExistsOnTheInternet/ToxicQAFinal (6,866 examples, 1 epoch). Compliant detailed answers paired with harmful prompts — teaches the model to answer instead of refusing.
• DPO corpus: unalignment/toxic-dpo-v0.2 (541 preference pairs, 1 epoch). chosen = compliant answer, rejected = refusal.
• LoRA: r=16, α=32, dropout=0.05.
• Target modules: (q,k,v,o)_proj, out_proj, (gate,up,down)_proj on text body layers (language_model.layers.*). Vision tower (visual.blocks.*) explicitly excluded via regex filter.
• Trainable params: 88.34M / 27.45B (0.322%).
• Optimizer: AdamW BF16, gradient checkpointing, effective batch 8 (per-device 2 × grad-accum 4), max_seq_len 1024.
• Compute: 1× NVIDIA H200 SXM 141GB, ~48 min total (SFT 41 min + DPO 4 min + setup overhead) with flash-linear-attention + causal-conv1d + tilelang fast path.
• Final DPO stats: train_loss 0.034, rewards/margins 14.4, rewards/accuracies 1.000 (100% preference accuracy on training set).

Stage 3 — Post-hoc drafter alignment

The HF wrapper Qwen3_5ForConditionalGeneration does not expose the MTP head as nn.Module, so PEFT could not train LoRA on it directly in the standard run. After body merge, the LoRA delta from body layer 63 (last full-attention layer, structurally matched to MTP) was projected onto the corresponding MTP weights via δ = (B @ A) × (α/r), applied to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. This is a heuristic but partially restores drafter / verifier distribution match. The MTP head retains its Stage 1 abliteration plus this projected delta.

What this means in practice

• The model rarely refuses on cybersec, social engineering, chemistry, and other subtle-harmful categories that the Jackrong distillation would normally refuse.
• Identity ("I am Qwen") is restored — the Claude-distillation identity artifact from the base is gone.
• Multimodal capability is preserved (vision tower never touched).
• Some MTP speculative-decode acceptance is lost relative to the pre-LoRA base; the decode-speed trade-off is detailed in the Benchmarks section below.

Model Details

Files

Quick CLI download (with huggingface_hub):

hf download croll83/Qwopus3.6-27B-v2-Abliterated-NVFP4 \
  Qwopus3.6-27B-v2-Abliterated-NVFP4.gguf \
  Qwopus3.6-27B-v2-Abliterated-mmproj-F16.gguf \
  --local-dir ./qwopus36-v2-abl

Quick start with llama.cpp (upstream stock)

The native MTP drafter (--spec-type draft-mtp) is in upstream ggml-org/llama.cpp since 2026-04 (PR #22673). NVFP4 MTP scale-tensor handling landed on 2026-05-23 (PR #23563). Build accordingly:

git clone https://github.com/ggml-org/llama.cpp ~/llama-cpp-stock
cd ~/llama-cpp-stock
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="120;121" \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-quantize llama-cli -j$(nproc)

CMAKE_CUDA_ARCHITECTURES="120;121" covers RTX 5090 (SM 12.0) and GB10 (SM 12.1) with native FP4 MMA. Drop 121 if you only target consumer Blackwell.

llama-server — production setup (port 30000)

./build/bin/llama-server \
  -m Qwopus3.6-27B-v2-Abliterated-NVFP4.gguf \
  --mmproj Qwopus3.6-27B-v2-Abliterated-mmproj-F16.gguf \
  --spec-type draft-mtp --spec-draft-n-max 5 \
  --host 0.0.0.0 --port 30000 \
  -c 262144 -np 2 -ngl 99 \
  -ctk q8_0 -ctv q8_0 \
  --slot-prompt-similarity 0.5 --cache-reuse 256 --cache-ram 16384 \
  -bs \
  --jinja --chat-template-file chat_template.jinja \
  --no-prefill-assistant --reasoning off \
  --alias dark-jarvis --alias dark-opus --no-webui --no-warmup

Key flags explained:

Inference temperature

Use temperature = 0.8 for general chat / reasoning / writing. The model is a reasoning-distilled fine-tune and tolerates higher temperatures than typical instruct models; below 0.5 it gets noticeably stiff. For deterministic tool/JSON output, use temperature = 0.0 + --json-schema.

Example OpenAI-compatible call:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "dark-jarvis",
    "messages": [{"role": "user", "content": "Explain reflection in 5 lines."}],
    "temperature": 0.8,
    "max_tokens": 500
  }'

llama-cli (one-shot)

./build/bin/llama-cli \
  -m Qwopus3.6-27B-v2-Abliterated-NVFP4.gguf \
  --mmproj Qwopus3.6-27B-v2-Abliterated-mmproj-F16.gguf \
  --spec-type draft-mtp --spec-draft-n-max 5 \
  -ngl 99 -c 8192 -ctk q8_0 -ctv q8_0 -bs \
  --jinja --temp 0.8 \
  -p "Hello"

Benchmarks (NVIDIA GB10, SM 12.1, 128 GiB unified mem)

Measured on dark-jarvis.service (this model, upstream stock, n_max=5, cache-ram 16 GiB, KV q8_0/q8_0, reasoning off). Real Hermes-agent workload over ~45 min (Markdown, code, JSON, structured writing, multimodal vision capture).

Decode throughput by workload type (single slot)

Peak sustained: 23.58 tok/s for 1 609 generated tokens, acceptance 85.6 % (code generation, single slot).

Prefill throughput

• Cold prefill (no cache): ~700 – 880 tok/s on 2 – 50 K-token prompts.
• Warm prefill (system prompt cached, only the new user message processed): TTFT ≈ 0.4 – 1.5 s regardless of system-prompt size.

Memory footprint (stable, no leak)

The GPU pool stays flat over hours of sustained traffic — no draft KV leak.

Reproducing the build

The text body uses plain NVFP4 (NVFP4_DEFAULT_CFG, no AWQ pre-quant scale) via NVIDIA ModelOpt, then the BF16 MTP-head weights from the original Jackrong checkpoint are merged into the same directory and a stock convert_hf_to_gguf.py produces the final GGUF.

High-level pipeline:

# 1. Source
hf download Jackrong/Qwopus3.6-27B-v2 --local-dir ~/qwopus36-v2-bf16

# 2. Abliterate (skip MTP head). See abliterate_qwopus_v2_mtp.py in repo.
python3 abliterate_qwopus_v2_mtp.py ~/qwopus36-v2-bf16 ~/qwopus36-v2-mtp-abl

# 3. NVFP4 quantize text body (HF does not expose MTP head to ModelOpt — that's fine)
python3 quantize_nvfp4_plain.py ~/qwopus36-v2-mtp-abl ~/qwopus36-v2-mtp-abl-nvfp4

# 4. Re-inject BF16 MTP weights from the original safetensors
python3 inject_mtp.py ~/qwopus36-v2-bf16 ~/qwopus36-v2-mtp-abl-nvfp4

# 5. Convert to GGUF with upstream stock script (PR #23563+)
cd ~/llama-cpp-stock
python3 convert_hf_to_gguf.py ~/qwopus36-v2-mtp-abl-nvfp4 \
  --outfile ~/qwopus36-v2-mtp-abl-nvfp4/Qwopus3.6-27B-v2-Abliterated-NVFP4.gguf

# 6. mmproj from BF16 source (vision tower unchanged by abliteration)
python3 convert_hf_to_gguf.py ~/qwopus36-v2-bf16 --mmproj --outtype f16 \
  --outfile ~/qwopus36-v2-mtp-abl-nvfp4/Qwopus3.6-27B-v2-Abliterated-mmproj-F16.gguf

Full reproduction guide: croll83/jarvis/infrastructure/gb10/dark-jarvis.md.

Important Disclaimers

This model has reduced safety filtering and may generate content that is sensitive, controversial, or potentially harmful.

• Intended for research and experimental use only.
• Not suitable for public-facing applications or use by minors.
• User is solely responsible for legal and ethical compliance.
• No default safety guarantees are provided.
• Use at your own risk and discretion.

Credits

• Base model: Jackrong/Qwopus3.6-27B-v2
• Foundation: Unsloth/Qwen3.6-27B → Qwen/Qwen3.5
• Abliteration technique: Sumandora/remove-refusals-with-transformers
• Harmful prompts: walledai/HarmBench
• Harmless prompts: Sumandora's harmless.txt (sampled from Alpaca-cleaned)
• MTP spec-decode in llama.cpp: PR #22673, PR #23461 (server VRAM fix), PR #23563 (NVFP4 MTP scale tensors)
• NVFP4 quantization: NVIDIA ModelOpt
• Inspiration: HuiHui-AI, z-lab/DFlash