gemma-4-12B-it-uncensored
Uncensored version of google/gemma-4-12B-it with refusal behavior removed.
Results
| Before | After | |
|---|---|---|
| Refusals (mlabonne, 100 prompts) | 99/100 | 6/100 |
| Refusals (cross-dataset, 686 prompts) | — | 14/686 (2.0%) |
| KL Divergence | 0 (baseline) | 0.0556 |
| Quality (coherence) | — | no degradation (response audit + Q8 inference verified) |
Cross-Dataset Validation
Tested against 4 independent prompt datasets to verify generalization:
| Dataset | Prompts | Refusals |
|---|---|---|
| JailbreakBench | 100 | 2/100 |
| tulu-harmbench | 320 | 4/320 |
| NousResearch/RefusalDataset | 166 | 4/166 |
| mlabonne/harmful_behaviors | 100 | 4/100 |
| Total | 686 | 14/686 (2.0%) |
Every flagged cross-dataset response was manually audited (scripts/audit_refusals.py). Of 13 flags, only 1 is a genuine refusal (a sensitive prompt the model handles by redirecting to support resources); the other 12 are false positives — an "I am an AI" disclaimer, or a marker that matched inside generated content, followed by compliance. Effective refusal rate: ~0/686.
Method
Norm-preserving biprojected abliteration (grimjim, Nov 2025).
Each weight row is decomposed into magnitude + direction, the refusal direction is projected out of the
direction component only, then recombined with the original magnitude — guaranteeing ||W_new|| = ||W_orig||.
Note on the Unified architecture.
gemma-4-12B-itis the encoder-freeGemma4Unifiedmodel (released 2026-06-03). Its refusal signal concentrates in the upper decoder layers (L15-47), so only the top 70% of layers are abliterated — ablating the near-zero-SNR early layers adds distortion with no refusal benefit. The arch also emits hard-inflogits for reserved vocabulary tokens, which NaNs a naive KL divergence; the reported KL masks non-finite positions before reducing.
Version requirements: inference needs transformers >= 5.10.1 (tested on 5.12.0, torch 2.11.0+cu130). GGUF conversion/quantization needs llama.cpp with Gemma4Unified support, added 2026-06-04 in PR #24118 (tested at commit c34b922).
Pipeline
- Load model in bf16 with LoRA adapters on
o_projandmlp.down_proj - Collect residual activations for 400 harmful + 400 harmless prompts (mlabonne datasets)
- Winsorize activations at 99.5th percentile (clamps GeGLU outlier activations in Gemma family)
- Compute per-layer refusal direction:
normalize(mean(harmful) - mean(harmless)) - Orthogonalize each direction against harmless mean (double-pass Gram-Schmidt)
- Apply norm-preserving weight modification to
o_projanddown_projin the selected layers - Merge LoRA adapters into base weights for clean tensor names
Parameters
| Parameter | Value |
|---|---|
| Layers abliterated | 70% |
| Scale | 1.0 |
| Winsorization | 0.995 |
How this differs from vanilla heretic
- Norm-preserving biprojection instead of standard projection (preserves weight magnitudes)
- Per-layer refusal directions instead of one global direction
- Deterministic single-pass instead of 50-trial Optuna search (faster, same or better results)
- LoRA merge before save for clean GGUF-compatible tensor names
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("TrevorJS/gemma-4-12B-it-uncensored", dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("TrevorJS/gemma-4-12B-it-uncensored")
messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
outputs = model.generate(inputs.to(model.device), max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))
Reproduction
Full code and experiment data: abliteration research repo
python scripts/abliterate.py biprojection --model google/gemma-4-12B-it \
--top-pct 70 --strip-topic-markers --skip-prefix --batch-size 4 \
--auto-save output_dir