Qwen3-Coder-30B-A3B-Audit

A Solidity security-audit model fine-tuned from Qwen/Qwen3-Coder-30B-A3B-Instruct on 10,821 real vulnerability findings adjudicated by human auditors in public audit contests.

Given a Solidity snippet, it returns a JSON array of security findings — severity, category, function, line range, and a one-line reason — or an empty array when the code is clean.

Fine-tuning cuts false positives by 75% and raises the clean-code pass rate from 5% to 73%. The base model flags a vulnerability in 95% of clean contracts; this model does not.

Serving

vllm serve qtum/Qwen3-Coder-30B-A3B-Audit --max-model-len 16384

A Mixture-of-Experts model with 30B total and ~3B active parameters — throughput close to a 3B dense model at the capability of a much larger one. bf16 weights, ~57 GB.

Prompt format

The model is trained for one specific system prompt. Deviating from it degrades output quality.

<|im_start|>system
You are a smart contract security auditor. You are given a Solidity code snippet taken from a
real audited codebase. The comment `// ... (第 A-B 行)` tells you the original file line numbers
of the snippet body.

Report only real, exploitable security vulnerabilities. Do NOT report style issues, gas
optimizations, or speculative concerns. Most snippets contain at most one or two real issues;
many contain none.

Respond with ONLY a JSON array. Each element must have exactly these fields:
  "severity": "High" | "Medium" | "Low"
  "category": one of the following slugs, nothing else:
- reentrancy
- access-control
- input-validation
- arithmetic
- accounting
- oracle
- mev-slippage
- dos
- liquidation
- rewards-fees
- token-integration
- signature-replay
- init-upgrade
- crosschain
- flashloan-donation
- callback-hook
- other
  "function": the name of the function containing the bug
  "lines": the original file line range of the bug, e.g. "412-418" (keep it tight, under 40 lines)
  "reason": one sentence explaining the vulnerability

If the snippet has no real vulnerability, respond with exactly: []
No prose, no markdown fences, no explanation outside the JSON array.<|im_end|>
<|im_start|>user
Audit this snippet:

```solidity
{contract_code}
```<|im_end|>
<|im_start|>assistant

Recommended sampling: temperature=0, max_tokens=4096, thinking disabled, one sample.

Greedy decoding is not just the default here — it is the best configuration we measured. Drawing five samples at temperature=0.7 and taking the union raises recall from 0.082 to 0.148 but pushes false positives from 0.27 to 2.76 per snippet; filtering that union by agreement across samples (keep findings seen in ≥2 of 5) lands at recall 0.079 / FP 0.38 — strictly worse than a single greedy sample on both axes. Vote count does carry a weak signal (hit rate rises from 3.3% at ≥1 vote to 10.2% at ≥4), but not enough to separate the extra findings from the noise.

Results vs. base model

Evaluated on two independent held-out sets built from audit contests the model never saw: EVAL-A (Code4rena, 377 questions / 305 gold findings) and EVAL-B (Sherlock, 770 / 715). Both sets include 30–35% clean contracts to measure false positives.

Confirmed findings rose from 4/305 to 25/305 on EVAL-A and 10/715 to 48/715 on EVAL-B — +59 hits pooled, 6.3 standard errors. Pooled across both sets the improvement is statistically significant.

The largest gains are on the false-positive side, and recall rises together with precision — the model is not simply reporting less.

Putting the numbers in context

The gold set is the union of every finding submitted by every participant in a contest — typically 37 auditors per contest. No individual reaches it. Measured across 155 Sherlock contests using the published "Found by" rosters:

On that scale the model performs roughly like a median contest participant rather than "9% of a perfect score". Three caveats, in both directions: the tasks are not equivalent and the difference favours the model (auditors work a whole codebase and must write PoCs; the model is handed a function-level snippet that contains the bug); the 12% median includes many low-effort submissions; and the semantic verifier used here is conservative — it rejects 23% of genuine paraphrases, so the true figure is likely 10–11%.

The meaningful target is not 100% but the ~59% a strong human auditor reaches.

Examples

Each example is verbatim output on a held-out contract the model never saw during training.

Oracle staleness — EthOracle.sol (bakerfi)

// gold:  [Medium / oracle]  Min and maxAnswer never checked for oracle price feed
// base:  wrong reason, over-rated severity
[{"severity":"High","category":"oracle","function":"getLatestPrice",
  "reason":"Missing check for stale price data ..."}]
// this model:
[{"severity":"Medium","category":"oracle","function":"getLatestPrice","lines":"20-44",
  "reason":"Chainlink's `latestRoundData` can return stale or incorrect results"}]

Signature replay — TimelockTokenPool.sol (Taiko)

// gold:  [High / signature-replay]  Signatures can be replayed in `withdraw()`
// base:  misclassified as an access-control issue
[{"severity":"High","category":"access-control","function":"withdraw",
  "reason":"The withdraw function allows anyone to specify any recipient address ..."}]
// this model:
[{"severity":"High","category":"signature-replay","function":"withdraw","lines":"158-183",
  "reason":"TimelockTokenPool#withdraw can be called multiple times"}]

Wrong exchange rate — xRenzoDeposit.sol (Renzo)

// gold:  [Medium / oracle]  Incorrect exchange rate provided to Balancer pools
// base:  flagged an unrelated function entirely
[{"severity":"High","category":"access-control","function":"setAllowedBridgeSweeper",
  "reason":"The function lacks proper access control validation ..."}]
// this model:
[{"severity":"Medium","category":"oracle","function":"getRate","lines":"446-468",
  "reason":"`xRenzoDeposit`'s `getRate()` function returns wrong value"}]

Knowing when to stay silent — PeripheryPayments.sol, a clean contract

// gold: []
// base: hallucinates a High-severity reentrancy
[{"severity":"High","category":"reentrancy","function":"unwrapWETH9",
  "reason":"... transfers ETH without checking if the recipient can reenter ..."}]
// this model:
[]

Training

Method — LoRA (rank 32, alpha 64, dropout 0.05) on attention projections (q_proj, k_proj, v_proj, o_proj); 26.7M trainable parameters, 0.0875% of the base. Loss is computed on the assistant response only. 2 epochs, lr 1e-4 cosine, sequence length 6144, bf16 on 2×H100. Adapter merged into the base weights for this release.

Data — 3,492 examples derived from public audit-contest findings:

Two design choices mattered more than hyperparameters:

• Negatives at 35%. Without clean-code examples the model reports a vulnerability in almost every snippet — the base model's clean-code pass rate is 0.05. This ratio was chosen to match the evaluation prior, and it is the single largest contributor to the false-positive reduction.
• Multi-finding targets. Snippets containing several adjudicated findings are trained as a multi-element array. An earlier version trained one finding per example and learned to never emit more than one, capping recall by construction.

Evaluation methodology

Numbers above come from a purpose-built benchmark rather than a generic code-security set.

Leak prevention, three gates. Evaluation uses only contests from 2024 onward; any protocol appearing on both sides of that boundary is removed entirely (16 protocols, 121 findings — a time-only split would have leaked); and every evaluation snippet is fingerprinted against the training corpus to catch forked code.

Scoring, three layers. A prediction counts only if (1) its function or line range overlaps the gold anchor, within a 40-line span cap, (2) its category matches the gold category from a controlled 17-slug vocabulary, and (3) a separate verifier model confirms the two descriptions refer to the same bug. The third layer removes roughly 60% of rule-level matches — the same function often contains more than one issue.

The scorer is itself tested, in both directions. An all-empty baseline must score 0; a hardcoded five-template baseline that never reads the code must score ≤0.10 (measured: 0.018); and feeding the gold answers back in must score ≥0.85 (measured: 0.888). The upper-bound test is not optional — it is what catches a scorer that silently under-counts correct answers.

What does not help

These were measured on the same held-out benchmark and are reported so you do not spend time re-discovering them. All comparisons use hit counts against 305 gold findings with Poisson standard errors; "no effect" means within two standard errors.

The consistent pattern is that extra context, extra samples, extra parameters and extra domain knowledge all fail to move the number. Ten configurations land between 0.055 and 0.087. That band appears to be the ceiling of the task as posed — one snippet in, one JSON verdict out, no iteration and no verification — rather than a limit of any particular model or dataset.

Limitations

This model assists human auditors; it does not replace them.

• Absolute recall is low. 25 of 305 gold findings on EVAL-A — about 9% of the ceiling a perfect answer would reach on the same metric. Treat its output as a triage signal.
• Roughly 0.25 false positives per snippet. Far below the base model, but every finding still needs human confirmation.
• Function-level snippets only. Trained and evaluated on single functions with surrounding context, not whole repositories or cross-contract call graphs. Vulnerabilities that only appear across contract boundaries are out of scope.
• Solidity / EVM only.
• Fixed output schema. The 17-category vocabulary and JSON shape are baked in by training.

Details

Fine-tuned by qtum. Training data is derived from publicly published smart-contract audit contest reports. Findings produced by this model are automated suggestions and are not a security guarantee; independent review is required before relying on them.