Qwen3.8-27B-Solidity-v1

• +23% F1 over the base model and +20% over DeepSeek V4 Pro (1.6T MoE, ~60× larger) on 191 held-out smart contracts.
• +49% precision over the base model: fewer false accusations, less noise per contract.
• Built from 3,225 Solidity contracts, 413 professional audits and 3,987 judged findings (2,998 training examples; the rest held out for testing).
• Custom imatrix GGUF mix (Q4_K_Mix): 43% less divergence than Q4_K_M + imatrix, and runs with 128k context on a single 32 GB GPU.

A LoRA fine-tune of Qwen3.8-27B trained on Solidity smart-contract audit data, so that it handles smart contracts better than the base model. The data combines findings from professional smart-contract security audits and analyses of real-world exploits, linked to the vulnerable contract code. The repo also contains a quantization study with custom imatrix-guided GGUF mixes, built to run the model on a single 32 GB V100.

Dataset in numbers (train + held-out validation/test):

• 3,225 Solidity contract files, each paired with its expected audit result: 2,998 for training, the rest held out for validation and test.
• 413 professional security audits of DeFi protocols.
• 3,987 judged High/Medium findings, each with root cause, attack path and fix, linked to the vulnerable code.
• 58 real-world exploits, with the victim contracts' verified source code.
• 560 clean contract files: in-scope files with no valid High/Medium finding after judging.
• 595 step-by-step reasoning traces from DeepSeek V4 Pro, kept only when they reached the same findings as the human auditors.
• About 13M tokens of training text.

This is a personal end-to-end project covering dataset construction, LoRA training, evaluation against the base model and a frontier model, quantization, and testing on real hardware.

Model details

Results at a glance

On a held-out test set of 191 contracts (split by protocol, so no project appears in both train and test), compared with the base model and DeepSeek V4 Pro, a 1.6T-parameter MoE model (49B active), about 60× larger than this 27B model:

On 37 real exploits from 2026 (incidents that happened after every model's training cutoff), F1 went from 0.37 (base) and 0.44 (DeepSeek) to 0.49 with the recommended quant (+32% vs base, +11% vs DeepSeek). The plain Q4_K_M build reached 0.58. With only 37 samples, differences of a few points may be noise.

General skills were mostly preserved: Spanish Q&A stayed at 16/16, and Python coding problems with tests went from 15/16 to 14/16.

Training

Each training example is one contract file plus the expected findings: title, severity, location, root cause, attack path, impact and fix. The training data is not published.

Training data

• Totals: 2,998 train / 36 validation / 191 test examples, split by protocol (305 / 9 / 60 protocols), so that no project appears in more than one split.
• Train size: about 13M tokens.
• Audit dates: 2022–2025.
• 2026 exploit test set: all 37 exploits from 2026 are kept in test only.

Evaluation details

Audit benchmark: 191 contracts, no thinking

The metrics are automatic and location-based. Precision is how many of the flagged functions are the real bug. Recall is how many of the real bugs are flagged.

Real 2026 exploits: 37 incidents

Thinking mode: 60 contracts, 20k-token budget

After fine-tuning, thinking mode finishes far more often than in the base model. It still scores worse than answering directly.

Quantization study

The imatrix was computed from a custom calibration set: Solidity 50%, Foundry tests 15%, ERCs 10%, Python 10%, Go/Rust 5%, Spanish 10%. KLD was measured against the BF16 model on six domains (lower = more faithful to the full-precision model).

How mix I (Q4_K_Mix-imat) is built:

• Q6_K on every sensitive tensor (ffn_down, attn_qkv, ssm_out, attn_output, attn_v, output).
• Q5_K on ffn_up/gate of the most important layers by imatrix statistics (51–63).
• Q4_K with imatrix on everything else.

At about 11% more size than Q4_K_M, it has 43% less divergence than Q4_K_M + imatrix.

Lessons learned:

• Adding --tensor-type rules on top of Q4_K_M shifts its internal "more bits" heuristic and silently drops some Q6_K tensors to Q4_K. Every sensitive tensor needs an explicit rule.
• Pushing the least important layers down to Q3_K to pay for upgrades elsewhere made the model worse than plain B.

Quick start

Download one file (the recommended mix):

huggingface-cli download Rami8612/Qwen3.8-27B-Solidity-v1-GGUF Qwen3.8-27B-Solidity-v1-Q4_K_Mix-imat.gguf --local-dir .

Run with llama.cpp (recent build, with support for the Qwen3.8 hybrid architecture):

llama-server -m Qwen3.8-27B-Solidity-v1-Q4_K_Mix-imat.gguf -ngl 99 -fa on -c 65536 --jinja --reasoning off

The server is OpenAI-compatible at http://localhost:8080/v1. The GGUF files also load in LM Studio and other llama.cpp-based apps.

Which file should I choose?

Only the I figure was measured. The other values are estimates.

Prompt format and settings

• Chat template: Qwen ChatML, embedded in the GGUF (use --jinja).
• Thinking: off (--reasoning off, or "chat_template_kwargs": {"enable_thinking": false}). In our tests the fine-tune works best without thinking.
• Temperature: 0 for repeatable output. 0.7–0.8 explores more varied answers, so running a few samples can surface issues a single greedy pass misses.
• Speculative decoding: no MTP / speculative decoding (see below).
• System prompt used in training and evaluation:

You are an expert smart contract security auditor performing an authorized security review. Review the Solidity code you are given and report every High or Medium severity vulnerability you find. For each finding give a title, severity, location (contract/function), the root cause, how an attacker would exploit it step by step (attack path), the impact, and a recommended fix. If there are no High or Medium issues, state that clearly.

• User message format used in training:

Audit the following Solidity file from the project `<project>`.

File: `<path>`

```solidity
<code>
```

Running it on a V100 32 GB

• Software: CUDA 12 (CUDA 13 dropped Volta) and llama.cpp built with -DCMAKE_CUDA_ARCHITECTURES=70.
• Fit: Q4_K_Mix-imat with 128k context and a q8_0 KV cache uses about 22.5 GB.
• Speed: prompt processing ~500–760 tok/s, generation ~30 tok/s.

llama-server -m Qwen3.8-27B-Solidity-v1-Q4_K_Mix-imat.gguf -ngl 99 -fa on -c 131072 -ctk q8_0 -ctv q8_0 \
  --jinja --reasoning off

• MTP speculative decoding (--spec-type draft-mtp) doubles the speed (65 tok/s) but degrades this fine-tune's output: repeated findings, or loops with an f16 cache. The MTP head (blk.64.nextn) is the untouched base-model head. Do not use it with this model.
• Thinking off is recommended.

Limitations

This is a v1 experiment. Tests on real-world contracts beyond the benchmark showed clear weaknesses:

• Findings on clean code: it reports findings on most clean files (86–97%), because only 17% of the training data were negatives.
• Hallucinations: it sometimes invents code, such as treating SafeERC20 library helpers like safeTransfer as token functions, and it can repeat the same finding.
• Complex bugs: it catches direct missing-check bugs well. Complex cross-function bugs (e.g. re-entrancy through an inherited entry point) appear only occasionally, and only with sampling (temperature ~0.8).
• Metrics: the metrics are automatic approximations.

Files

Make your own quant, e.g. Q6_K:

llama-quantize --imatrix Qwen3.8-27B-Solidity-v1-imatrix.gguf Qwen3.8-27B-Solidity-v1-BF16-00001-of-00002.gguf Qwen3.8-27B-Solidity-v1-Q6_K-imat.gguf Q6_K

Compute

Changelog

• v1 (2026-10-03): first release.
  • LoRA SFT on Solidity audit data;
  • custom imatrix GGUF mixes;
  • evaluation against the base model and DeepSeek V4 Pro.

Citation

@misc{rami8612_qwen38_solidity_sft_v1,
  title  = {Qwen3.8-27B-Solidity-v1: fine-tuning and quantization study},
  author = {Rami8612},
  year   = {2026},
  url    = {https://huggingface.co/Rami8612/Qwen3.8-27B-Solidity-v1-GGUF}
}

Credits and license

• Base model: unsloth/Qwen3.8-27B, Unsloth's release of Qwen3.8-27B by the Qwen team (Apache-2.0).
• Reasoning traces: generated with DeepSeek V4 Pro (MIT).
• Training data: derived from publicly available smart-contract audit reports, exploit analyses and contract source code.
• Tools: training with Unsloth and TRL; quantization and serving with llama.cpp.