Hirundo Gemma 4 E4B IT Hardened
This repository contains a behaviorally hardened derivative of google/gemma-4-E4B-it. Hirundo modified the model to reduce prompt-injection susceptibility while preserving its general capabilities.
The checkpoint itself changed; this is not a runtime prompt, output filter, or external guardrail.
About Hirundo
Hirundo develops machine-unlearning technology to remove unwanted data and reduce unwanted behaviors in trained AI models. Its behavioral-unlearning workflow identifies a target behavior, modifies the checkpoint, and evaluates the result on both behavior-specific and general-capability benchmarks.
Learn more:
What Hirundo changed
The target behavior was prompt injection: adversarial or untrusted instructions that cause a model to disregard its intended task or reveal protected context. Hirundo applied behavioral unlearning to the Gemma 4 E4B instruction-tuned checkpoint with this run configuration:
| Field | Value |
|---|---|
| Base model | google/gemma-4-E4B-it |
| Target behavior | Security (prompt injection) |
| Unlearning aggressiveness | 0.1 |
Evaluation summary
Hirundo compared the hardened checkpoint with the unmodified base checkpoint with thinking mode off. The reported prompt-injection and general-capability evaluations do not establish security or utility performance for image, audio, or video inputs. For the security evaluations below, lower scores indicate fewer successful attacks or leaks and therefore better performance. Relative reduction is (base − hardened) / base × 100.
Headline result: PurpleLlama prompt injection
PurpleLlama's Cybersecurity Benchmarks measure prompt-injection attack success. The measured attack success rate (ASR) fell from 18.73% to 4.78% in this evaluation.
| Benchmark | Base model ASR | Hirundo model ASR | Relative reduction |
|---|---|---|---|
| PurpleLlama prompt injection | 18.73% | 4.78% | 74.47% |
General-capability evaluation
Six general-capability benchmarks were measured with thinking mode off. The table shows only the change from the base checkpoint, calculated as hardened minus base in percentage points. Positive values indicate improvement. In this evaluation, prompt-injection susceptibility decreased while general capabilities were broadly preserved.
| Benchmark | Change vs. base model |
|---|---|
| AIME25 | −0.83 pp |
| GPQA | +0.76 pp |
| IFBench | −0.26 pp |
| LiveCodeBench | −0.20 pp |
| MMLU-Pro | −0.36 pp |
| SciCode | 0.00 pp |
The unweighted mean change is −0.15 pp. The largest measured decrease is 0.83 pp on AIME25. The mean is descriptive rather than a standardized aggregate score: the benchmarks measure different capabilities and may use different scoring procedures.
Intended use
This checkpoint is intended for:
- mitigation research on prompt injection;
- comparative security evaluation against the base model;
- development of applications that need a hardened checkpoint; and
- further evaluation or fine-tuning for a specific deployment.
Test the checkpoint in the application's actual prompt, tools, retrieval pipeline, and threat model before deployment. These benchmark results do not establish immunity to prompt injection or guarantee the same performance in a different setting. The upstream Gemma 4 E4B model accepts text, image, audio, and video input and generates text; the evaluations above do not assess every modality. For architecture, supported languages, context length, generation parameters, and base-model considerations, see the upstream Google model card.
Loading with Transformers
The upstream multimodal model uses AutoProcessor and AutoModelForMultimodalLM. The example below uses a text prompt; follow the upstream usage examples for image, audio, and video inputs. Use a compatible Transformers release and suitable hardware.
from transformers import AutoProcessor, AutoModelForMultimodalLM
model_id = "hirundo-io/gemma-4-E4B-it-hardened"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
model_id, dtype="auto", device_map="auto"
)
messages = [{"role": "user", "content": "Hello!"}]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
enable_thinking=False,
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
response = processor.decode(
outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=False
)
print(processor.parse_response(response, prefix=inputs["input_ids"]))
License
This derivative is released under the repository's applicable license. The upstream Gemma 4 model is listed under Apache 2.0; review its license and usage terms before deployment.
Citation
@misc{hirundo_gemma_4_e4b_it_hardened,
title = {Hirundo Gemma 4 E4B IT Hardened},
author = {Hirundo},
howpublished = {\url{https://huggingface.co/hirundo-io/gemma-4-E4B-it-hardened}}
}