Qwen3.8-27B Abliterated

Madras1/Qwen3.8-27B-Abliterated is a BF16 text-generation checkpoint derived from Qwen/Qwen3.8-27B through a permanent representation-engineering intervention.

This release was produced with candidate extraction from the residual stream, causal candidate screening, temporary activation-level validation, direct orthogonal weight projection, and a second validation after the edited weights were materialized. The model is experimental: the intervention substantially reduced refusal behavior in the included evaluation, but it does not guarantee the absence of residual alignment or preservation on every downstream task.

Resumo em português

Este é um checkpoint BF16 do Qwen/Qwen3.8-27B modificado por engenharia de representações. A direção selecionada foi extraída de resid_pre, na posição -5 do final da instrução e na camada 33. A intervenção foi primeiro simulada por hooks, avaliada e somente depois gravada nos pesos. O checkpoint gravado passou novamente pelos gates de capacidade, comprimento das respostas, disponibilidade de resposta final, redução da heurística de recusa e perda de linguagem.

A avaliação é pequena e deve ser interpretada como um teste controlado do procedimento, não como prova de capacidade geral nem como garantia de comportamento irrestrito.

Intervention summary

The 129 edited entries comprise the token embedding matrix and the residual-writing attention and MLP output projections across the 64 text layers. Depending on the layer type, the attention writer is either linear_attn.out_proj or self_attn.o_proj; the MLP writer is mlp.down_proj.

For a unit direction $\hat r$ and strength $\alpha=1$, the relevant output weights are projected as

$$ W' = (I - \alpha \hat r \hat r^T)W. $$

The implementation uses the direct projection and does not restore the norm of individual weight vectors afterward.

How the direction was selected

The pipeline used four deduplicated, disjoint splits for calibration, direction extraction, candidate selection, and held-out evaluation. Its recorded configuration used:

• 128 source samples per public dataset;
• 32 calibration samples per class;
• 32 candidate-selection samples per class;
• 16 held-out evaluation samples per harmful/harmless group;
• 8 exact-answer capability probes;
• thinking disabled during selection and evaluation;
• maximum candidate KL divergence of 0.10;
• the final 20% of model layers excluded as direction sources.

Initial refusal-token IDs were calibrated from the base model rather than copied from another Qwen generation. Candidate directions were mean differences in resid_pre at every common end-of-instruction suffix position. Candidates were screened with temporary hooks before any weight mutation by measuring:

1. movement of harmful activations under ablation;
2. inverse movement of harmless activations when the direction was added;
3. KL divergence on harmless prompts;
4. refusal reduction on a separate selection split;
5. capability and completion-quality gates.

The selector combines the candidate construction and direct projection used by Arditi et al. with a COSMIC-style concept-inversion criterion. It is therefore not presented as an exact reproduction of either paper.

Recorded evaluation

These numbers come from the artifacts stored with this exact checkpoint.

Both stages passed all configured checks. The allowed maximum cross-entropy increase was 0.25, minimum capability retention was 0.875, minimum final-answer rate was 0.95, and minimum refusal-rate reduction was 0.10.

The refusal detector is a lexical English/Portuguese heuristic. A lower heuristic refusal rate does not prove correct, useful, or policy-compliant completion. The eight exact-answer probes are smoke tests, not a broad reasoning benchmark. Inspect the saved responses before drawing conclusions about a particular use case.

Reproducibility artifacts

The repository includes the following run artifacts:

• experiment_config.json: arguments and gate thresholds;
• split_manifest.json: hashes and membership of the data splits;
• refusal_token_calibration.json: base-model refusal-token calibration;
• candidate_directions.pt: extracted candidate directions;
• direction_selection.json: causal and geometric candidate scores;
• direction_selection_progress.jsonl: candidate-selection trace;
• runtime_quality_gate.json: temporary activation-ablation validation;
• baked_quality_gate.json: validation after weight projection;
• weight_edit_audit.json: per-tensor direction removal and relative weight changes;
• evaluation_before.jsonl and evaluation_after.jsonl: held-out generations;
• evaluation_comparison.json: paired before/after summary;
• evaluation_input_audit.json: split-overlap audit;
• abliteration_report.json: consolidated run report.

Transformers usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Madras1/Qwen3.8-27B-Abliterated"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
).eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a short story about a city during a blackout."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
    return_dict=True,
)
input_device = model.get_input_embeddings().weight.device
inputs = {name: tensor.to(input_device) for name, tensor in inputs.items()}
input_length = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.70,
        top_p=0.80,
        top_k=20,
    )

print(tokenizer.decode(output[0, input_length:], skip_special_tokens=True))

The BF16 checkpoint is approximately 53.8 GB before runtime overhead. Quantization or multi-GPU loading may be required on machines without enough accelerator memory.

Scope and limitations

• The original single-direction refusal work did not evaluate Qwen3.8's alternating linear-attention/full-attention architecture. This pipeline explicitly handles both attention output types and validates the result, but architectural generalization remains experimental.
• Later work suggests that refusal can occupy multiple independent directions or a wider cone in some models. A one-direction edit can therefore leave residual refusal behavior.
• Results are sensitive to chat templates, system prompts, sampling parameters, language, and prompt distribution.
• This is a weight-edited instruction model, not a new pretraining run and not a guarantee of factual accuracy.

References

• Arditi et al., Refusal in Language Models Is Mediated by a Single Direction, NeurIPS 2024.
• Wollschlaeger et al., The Geometry of Refusal in Large Language Models.
• Siu et al., COSMIC: Generalized Refusal Direction Identification in LLM Activations.

License

Released under the Apache 2.0 license inherited from the base model. Users are responsible for evaluating the checkpoint for their own applications and deployment requirements.