LFM2.5-350M-Abliterated
Abliterated (refusal-vector-ablated) version of LiquidAI/LFM2.5-350M.
Original model: ~88% refusal rate on HarmBench This model: 25.3% refusal rate (74.7% compliance)
⚠️ Disclaimer
This model has been modified to reduce safety-related refusals. It may generate content that the original model would refuse, including harmful, dangerous, unethical, or illegal content. Use at your own risk. This model is intended for safety research, red-teaming, and understanding refusal mechanisms in language models. It should not be deployed in production or user-facing applications without additional safeguards.
This is not a perfect abliteration. Approximately 25% of harmful prompts are still refused. Further optimization (per-component alpha tuning, Optuna-based parameter search, soft convolution ablation) could reduce this further. Improvements are possible — this represents a snapshot of current methodology.
Method
Magnitude-Preserving Orthogonal Ablation (MPOA) with per-layer float-direction interpolation, targeting attention output projections only.
| Parameter | Value |
|---|---|
| Target matrices | self_attn.out_proj (6 GQA layers) |
| Ablation strength (alpha) | 2.5 |
| Direction offset (t) | 0.8 (float-interpolated between adjacent layer directions) |
| Direction source | 50 harmful + 50 benign prompts, SVD-whitened difference-of-means |
| Conv layers | Not modified (too sensitive to ablation) |
| FFN layers | Not modified (preserves factual knowledge) |
Architecture Note
LFM2.5-350M uses a hybrid architecture: 6 GQA (Grouped-Query Attention) layers interleaved with 10 LIV (Liquid Convolution) layers across 16 total decoder layers. Only the 6 attention output projections were modified. Convolution layers were left untouched because they are significantly more sensitive to weight perturbation — even small alphas (>=0.3) cause immediate token collapse.
Why Per-Layer Float-Direction Interpolation
Standard abliteration uses a single refusal direction for all layers. This fails on LFM2.5 because each layer processes different levels of abstraction — a direction from layer 12's hidden state cannot effectively ablate layer 2's output. We use each layer's own refusal direction computed from its hidden state output, with float-direction interpolation (t=0.8) blending toward the next layer's direction. This captures the evolution of the refusal signal across layers.
Performance
Evaluated on the full HarmBench DirectRequest test set (320 prompts):
| Metric | Original | Abliterated |
|---|---|---|
| Refused | ~88% | 25.3% (81/320) |
| Complied | ~12% | 74.7% (239/320) |
| Garbled | 0% | 0% |
74.7 percentage-point reduction in refusal rate with zero generation quality degradation. All compliant outputs are well-formed English prose.
Quality Caveat
While all outputs are structurally coherent, approximately 15-25% of compliant responses show mild content degradation (off-topic drift, hallucinated details in creative prompts). This is expected — in a 350M-parameter model, refusal direction and general language capability share representation space.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "PinoCookie/LFM2.5-350M-abliterated"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
messages = [{"role": "user", "content": "Who are you?"}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Limitations
- 25% residual refusal: Some harmful categories (chemical synthesis, exploit code, physical violence) still trigger refusals. Per-component optimization or Optuna-based parameter search could reduce this further.
- Content degradation: Some content of the model might have degraded, because of the changes made.
- Conv layers untouched: 10 of 16 layers (LIV convolution) were not modified. Soft ablation at very low alpha (0.05-0.10) may help but requires careful tuning.
- Single alpha: All attention layers use the same ablation strength. Per-layer tuning based on separation scores (which range from 3.85 to 5.20) could improve results.
Citation
@software{lfm25_350m_abliterated,
author = {PinoCookie},
title = {LFM2.5-350M-Abliterated},
year = {2026},
url = {https://huggingface.co/PinoCookie/LFM2.5-350M-abliterated},
}
Based on LiquidAI/LFM2.5-350M by Liquid AI. Abliteration methodology based on Heretic and Arditi et al. 2024.