Hirundo Granite-4.1-3B Hardened
This repository contains a behaviorally hardened derivative of ibm-granite/granite-4.1-3b. Hirundo modified the model to reduce prompt-injection susceptibility while preserving its general capabilities.
The checkpoint itself was modified at the model level. The intervention is not a runtime prompt, output filter, or external guardrail.
About Hirundo
Hirundo develops machine-unlearning methods for changing specific model behaviors. In its behavioral-unlearning workflow, a target behavior is identified, the model is modified, and the resulting checkpoint is evaluated for the targeted security behavior and general capabilities. This card reports the measured comparison for this checkpoint rather than a guarantee about every use case.
Learn more:
What Hirundo changed
Prompt injection occurs when adversarial or untrusted instructions cause a model to disregard its intended task or reveal protected context. Hirundo applied behavioral unlearning to reduce this susceptibility in the base checkpoint.
| Field | Value |
|---|---|
| Base model | ibm-granite/granite-4.1-3b |
| Target behavior | Security (prompt injection) |
| Unlearning aggressiveness | 0.1 |
Evaluation summary
The hardened checkpoint was compared with the unmodified base checkpoint under the reported evaluation setup, with thinking mode off. For the security evaluations below, lower scores indicate fewer successful attacks or leaks and therefore better performance. Relative reduction is (base − hardened) / base × 100 when the base rate is nonzero.
Headline result: PurpleLlama prompt injection
The PurpleLlama Cybersecurity Benchmarks provide the prompt-injection evaluation used here. Attack success rate (ASR) fell from 40.24% to 8.77%, a 78.22% relative reduction.
| Benchmark | Base model ASR | Hirundo model ASR | Relative reduction |
|---|---|---|---|
| PurpleLlama prompt injection | 40.24% | 8.77% | 78.22% |
Garak security evaluation
The Garak scanner probes models for several attack and leakage behaviors. Attack-family rates and leakage-oriented rates below have different definitions and should be read separately.
Results by attack family
| Attack family | Base rate | Hirundo rate | Relative reduction |
|---|---|---|---|
| Encoding attacks | 8.3800% | 0.0500% | 99.40% |
| Latent injection | 0.3300% | 0.1000% | 69.70% |
| Direct prompt injection | 0.2900% | 0.0000% | 100.00% |
The unweighted mean relative reduction across these three attack families is 89.70%.
Leakage-oriented results
The first three metrics report leaks per try or attempt. The attack success rate is measured per attack or prompt, which may include multiple tries; these rates therefore have different denominators.
| Metric | Base rate | Hirundo rate | Relative reduction |
|---|---|---|---|
| Any leak rate | 2.2502% | 0.0378% | 98.32% reduction |
| Exact value leak | 0.1815% | 0.0330% | 81.82% reduction |
| Exact key leak | 2.0781% | 0.0333% | 98.40% reduction |
| Attack success rate | 7.4187% | 0.1892% | 97.45% reduction |
The unweighted mean relative reduction across these four leakage metrics is 94.00%.
General-capability evaluation
Six general-capability benchmarks were compared with the base checkpoint. The table gives only the hardened checkpoint's change versus base in percentage points (hardened − base); positive values indicate improvement. The measured security improvement was accompanied by small changes on most of these benchmarks, although GPQA, IFBENCH, LIVECODEBENCH, and SCICODE declined.
| Benchmark | Change vs. base model |
|---|---|
| AIME25 | 0.00 pp |
| GPQA | −0.95 pp |
| IFBENCH | −0.90 pp |
| LIVECODEBENCH | −1.11 pp |
| MMLU-PRO | +0.16 pp |
| SCICODE | −0.31 pp |
The unweighted mean change is −0.52 pp. The largest measured decrease is 1.11 pp on LIVECODEBENCH. The mean is descriptive rather than a standardized aggregate score: the benchmarks measure different capabilities and may use different scoring procedures.
Intended use
This checkpoint is intended for:
- Research into mitigation of prompt-injection susceptibility through model modification.
- Comparative security evaluation against the unmodified base model and other defenses.
- Development of applications that need a model-level hardening component.
- Further evaluation and task-specific fine-tuning.
Test the checkpoint in the application's actual prompts, tools, retrieval pipeline, and threat model. Benchmark results do not establish immunity to prompt injection or guarantee that the measured improvements transfer to every deployment setup.
Granite-4.1-3B is a text-generation model supporting conversational text inputs. Consult the upstream model card for architecture, modalities, languages, context length, generation settings, and base-model considerations.
Loading with Transformers
Use a recent Transformers version with Granite 4.1 support. The causal-language-model loader and tokenizer follow the upstream model's example:
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "hirundo-io/granite-4.1-3b-hardened"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [{"role": "user", "content": "Who are you?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
License
This checkpoint is released under the repository's Apache-2.0 license. See the upstream model card for the base model's license and terms.
Citation
@misc{hirundo_granite_41_3b_hardened,
title = {Hirundo Granite-4.1-3B Hardened},
author = {Hirundo},
howpublished = {\url{https://huggingface.co/hirundo-io/granite-4.1-3b-hardened}}
}