mulita-extractor-qwen2.5-1.5b

A 1.5B model fine-tuned to extract structured findings from vulnerability scanner PDF reports (OpenVAS, Qualys, Nessus, ZAP). It is the embedded local model of MulitaMiner: the tool segments a report into finding blocks and the model turns each block into JSON (description, solution, severity, references, instances, ...).

This release is training run v4, served in MulitaMiner as mulita-qwen2.5-1.5b-v4.

Files

Usage

Register it with the Modelfile, not the GGUF alone. Without the TEMPLATE it carries, Ollama falls back to the template embedded in the GGUF and the model silently leaves references, instances and plugin empty.

hf download INARI18/mulita-extractor-qwen2.5-1.5b --local-dir mulita
cd mulita
ollama create mulita-qwen2.5-1.5b-v4 -f Modelfile

Then, in MulitaMiner:

mulitaminer experiment <reports> --models mulita-qwen2.5-1.5b-v4

The model expects MulitaMiner's prompts and block format, and was evaluated with json_schema constrained decoding on. Outside that pipeline it is an ordinary Qwen2.5-1.5B-Instruct with a strong bias toward this task.

Training

• Base: Qwen/Qwen2.5-1.5B-Instruct (via unsloth/Qwen2.5-1.5B-Instruct).
• SFT with QLoRA 4-bit (Unsloth + TRL), r=32, alpha=32, all attention and MLP projections; loss masked to the assistant turn; 2 epochs, lr 2e-4, effective batch 16, max sequence 12288.
• Data: 6841 findings from 136 reports (OpenVAS 5459, Qualys 910, Nessus 383, ZAP 89), 8740 examples. Each example is the exact production conversation: the tool's scanner prompt, report blocks, and the target JSON. Every finding appears twice, once as a single block and once inside the multi-block chunk the tool sends in production.
• Labels come from each scanner's own machine export (CSV, HTML, XML) paired to the PDF blocks, never hand-written. Text the PDF does not render was trimmed from the labels so the model is not taught to hallucinate it.
• Reports are lab scans of intentionally vulnerable public Docker images. The dataset is not published.

Evaluation

Held-out set: 8 reports (1028 findings) never seen in training, with their applications' hosts excluded from training across all scanners. Gold labels are the scanners' machine exports. Each model ran 5 times over the set.

Each cell is score (fill rate):

• recall: share of the 1028 gold findings the model extracted and that were matched to one. All other rows are scored on those matched pairs only.
• score: mean over the matched pairs, 0 to 1. A field the model left empty while the gold has it scores 0, and so does a field the model filled while the gold is empty. Pairs where both are empty are left out, and a literal N/A or Not applicable counts as empty on both sides.
• fill rate: share of matched pairs where the model filled the field.

• token F1: word overlap between extracted and gold text (lowercased, word order ignored), harmonic mean of precision and recall.
• set F1: F1 over the reference list, each item matched exactly after normalization.
• exact match: 1 if the normalized value equals the gold, else 0.
• structural: list of instances paired by URL, sub-fields scored by token F1, divided by the longer list so missing or extra items cost score.

How to read it: when this model fills a text field it copies the report almost verbatim (token F1 0.98 to 1.00 on the pairs it filled, against 0.81 to 1.00 for DeepSeek, which paraphrases). Its lower scores come almost entirely from omissions, fields left empty while the gold has them. DeepSeek omitted nothing.

Setup: RTX 5080, Ollama 0.34.0, this Modelfile, temperature 0. Extraction is deterministic in a fixed environment (the 5 runs were byte-identical), but changing the Ollama version moved instances and plugin by a few points, so pin the runtime when comparing numbers.

Limitations

• Misses about 3% of findings (recall 0.971) and leaves text fields empty more often than a large cloud model. That omission is the main gap, on every text field.
• Trained on four scanners only.
• Trained and evaluated on English reports only.

License

Apache 2.0