RedSage-K-SFT

Model summary

RedSage-K-SFT is an 8B cybersecurity model for translating natural-language requests into Kali/Linux commands. It is fine-tuned from RedSage-Qwen3-8B-Ins using supervised fine-tuning on KaliBench. It corresponds to “RedSage-K (SFT)” on our paper and project page.

Training

KaliBench contains 8,504 verified query-command pairs spanning 1,642 sub-tools and 23 capability dimensions. SFT uses 3,504 training pairs, each presented in three modes, for 10,512 examples. The remaining 5,000 pairs form the test split.

Training applies response-only negative log-likelihood to reference commands, without reasoning traces. Appendix G.2 reports:

See the training guide for reproduction. Set --max-seq-length 16384 to match the paper; the script defaults to 12,288.

Evaluation

Table 1 results on the 5,000-example KaliBench test split, in percent:

Average Total Score: 77.4%, up from 71.7% for RedSage-Ins (+5.7 percentage points).

Total Score averages tool accuracy, optional-argument F1, and positional-argument F1. Exact match uses canonicalization and alias-aware scoring. Evaluation uses vLLM in BF16, temperature 0.2, and a 16,384-token budget (8,192 input + 8,192 output), without thinking. See the evaluation guide for the full protocol.

Usage

pip install "transformers==4.57.6" accelerate torch safetensors

Load the merged checkpoint directly. Before publication, replace the model ID with its local directory.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "RISys-Lab/RedSage-K-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
).eval()

messages = [
    {"role": "system", "content": (
        "Translate the request into a single accurate Kali/Linux command. "
        "Return only <output>command</output>."
    )},
    {"role": "user", "content": "In list mode, display the privileges of user 'eve' as they would apply to the command 'cat /etc/shadow', using non-interactive mode."},
]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs, max_new_tokens=256, do_sample=False, use_cache=True,
        temperature=0.2,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
completion = outputs[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(completion, skip_special_tokens=True))
# It should return: "<output>sudo --list --user eve --command 'cat /etc/shadow' --non-interactive</output>"

Precision: Stored tensors are BF16, while config.json declares FP16. The example explicitly loads FP16; use torch.bfloat16 on compatible hardware to match the paper's evaluation precision.

Intended use and limitations

Designed for cybersecurity research, education, and command assistance in authorized environments.

• Commands may contain incorrect tools, flags, or arguments. Review them before execution; behavior also depends on tool versions and the local environment.
• KaliBench measures single-command generation, not execution success or multi-step agent performance. Verification can accept environment-related runtime failures and timeouts.
• Synthetic labels may contain errors, and alias-aware scoring may miss valid alternatives. Training and test sets share tools.
• Results do not establish multilingual, long-context, general-chat, or misuse-resistance performance.

Citation

If you use RedSage-K-SFT, please cite:

@inproceedings{li2026kalibench,
  title={KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards},
  author={Pengfei Li and Naufal Suryanto and Sicheng Zhang and Muzammal Naseer},
  booktitle={The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track},
  year={2026},
  url={https://openreview.net/forum?id=BUajyUxKK6}
}