SigmaForge Rule Generator
Fine-tuned Sigma detection-rule generator based on
Qwen/Qwen2.5-Coder-1.5B-Instruct. The model drafts and repairs
Sigma YAML from natural-language detections, sample
events, or broken rules, then downstream tools compile the YAML to Splunk SPL,
Elastic Lucene, and Microsoft Sentinel KQL.
Published by the Aria AI Security Research Team.
Training
| Property | Value |
|---|---|
| Base model | Qwen/Qwen2.5-Coder-1.5B-Instruct |
| Method | LoRA (r=16, α=32) then merge |
| Data | sigmaforge-detection-rules |
| Samples seen | 7,166 (6 ZeroGPU chunks) |
| Last train loss | 0.73 |
| Compute | Hugging Face ZeroGPU (sigmaforge-trainer) |
The supervised mix covers three dataset configs:
description_to_sigma— analyst description → Sigma YAMLsigma_repair— broken rule + validation errors → corrected rulelogs_to_sigma— positive/negative events → expected rule
Held-out evaluation (n=60, description_to_sigma test)
| Metric | Score |
|---|---|
| Valid YAML rate | 1.00 |
| Sigma schema pass rate | 0.92 |
| Splunk SPL compilation rate | 0.83 |
| Elastic Lucene compilation rate | 0.83 |
| ATT&CK mapping precision | 0.60 |
| ATT&CK mapping recall | 0.08 |
| ATT&CK mapping F1 | 0.14 |
The generator is reliable at emitting schema-valid, compilable Sigma YAML. Technique tagging is weaker: the model often omits ATT&CK IDs even when the rule body is usable. Pair it with the retrieval + pySigma validation loop in the SigmaForge Space rather than treating raw model output as a deployable rule.
Raw numbers: eval_results.json.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "alirezaaminzadeh/sigmaforge-rule-generator"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
messages = [
{
"role": "system",
"content": "You are SigmaForge. Write valid Sigma YAML only.",
},
{
"role": "user",
"content": (
"Write a Sigma rule for the following detection.\n\n"
"Detection: Suspicious access to LSASS memory from a non-system process\n"
"Log source: product=windows, category=process_access\n"
"MITRE ATT&CK: T1003.001"
),
},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
Validate every generated rule with pySigma before deploying it to a SIEM.
Related
- Dataset: sigmaforge-detection-rules
- Demo: sigmaforge
- Trainer: sigmaforge-trainer
- Collection: SigmaForge
License
MIT for this fine-tune. Upstream SigmaHQ rules remain under the Detection Rule License (DRL 1.1).