LateOn-Code-edge-EVMScout-17M

[!CAUTION] This is a research checkpoint, not an autonomous auditor or a safety certificate. On the locked unseen-family test, each trained run placed 11 or 12 of 63 known affected functions in the top 10. The frozen base placed 7. The improvement is repeatable, but absolute recall remains low.

The downloadable checkpoint is a 16.95M-parameter LateOn-Code-edge retriever. It contains no Gemma weights. Historical system experiments used a frozen Gemma reviewer only after retrieval. Ignore Hugging Face's generic SentenceTransformer(...) example; use MultiVectorEncoder(...) and MeanMaxSim below.

Abstract

This work studies whether an ultra-small late-interaction model can learn repository-scale vulnerability localization under a 4 GB GPU constraint. The retriever receives a general security mechanism and ranks every implemented function in a Solidity repository. A separate reasoning model can review the retrieved shortlist, while executable tools remain responsible for confirmation.

The released 16.95M-parameter checkpoint was fully fine-tuned from lightonai/LateOn-Code-edge with repository-listwise supervision and same-repository distractors. Evaluation was frozen across three seeds and conducted on unseen protocol families. Mean Recall@10 increased from 11.1% for the frozen base to 18.0%, and median affected-function rank improved from 98 to 43.7. These results support the narrow hypothesis that small specialized retrievers can improve review prioritization; they do not establish practical autonomous auditing capability.

Model description

Research question

Auditors do not choose one vulnerable function from four easy options. They decide where to start in a repository with hundreds of functions.

We test one hypothesis: can an ultra-small model trained on whole-repository rankings prioritize known affected functions better than its frozen code-retrieval base on previously unseen protocol families?

The locked experiment produced three consistent observations:

• The median affected-function rank moved from 98 to 42–45 in repositories with a median of 443 candidate functions.
• The number of affected functions in the top 10 moved from 7 of 63 to 11, 11, and 12 of 63 across three runs.
• Every run improved on both held-out sets.

The checkpoint met the predeclared improvement criterion but remained far below the stronger competitiveness thresholds. It is released to make the positive and negative results reproducible and to support follow-up work on data quality, repository context, and executable verification.

Try the localizer

pip install "sentence-transformers>=6.0.0" torch

import torch
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("starknet-ai/LateOn-Code-edge-EVMScout-17M")

query = "Find reentrancy or callback paths where an external call happens before critical state is updated."
functions = [
    """function withdraw(uint256 amount) external {
        require(balance[msg.sender] >= amount);
        (bool ok,) = msg.sender.call{value: amount}("");
        require(ok);
        balance[msg.sender] -= amount;
    }""",
    """function balanceOf(address user) external view returns (uint256) {
        return balance[user];
    }""",
]

query_tokens = model.encode_query([query], convert_to_numpy=False)[0]
document_tokens = model.encode_document(functions, convert_to_numpy=False)

# EVMScout 17M was trained with mean MaxSim.
scores = torch.stack([
    (query_tokens @ document.T).max(dim=1).values.mean()
    for document in document_tokens
])

print(scores.argsort(descending=True).tolist())
# The state-after-call withdraw path should rank first.

Scan a repository

The repository includes the scanner and the 12 security queries used in the demo:

python scan_multivector_repo.py /path/to/authorized/solidity/repo \
  --checkpoint starknet-ai/LateOn-Code-edge-EVMScout-17M \
  --output scan.json \
  --top-k 10

The scanner extracts each function and its nearby calls, modifiers, inheritance, and shared state. It scores each risk separately, then merges the rankings. Compare scores only within the same query.

System architecture

Solidity repository
      │
      ├── deterministic security slices
      │     ├─ modifiers and inheritance
      │     ├─ callers and callees
      │     ├─ state reads and writes
      │     └─ shared-state neighbors
      │
      ├── LateOn-Code-edge EVMScout 17M (released checkpoint)
      │     └─ top functions for 12 general mechanisms
      │
      ├── optional external hypothesis reviewer
      │
      └── Foundry / Slither / Echidna / Halmos
            └─ CONFIRMED / SUPPORTED / LEAD / REJECTED / UNRESOLVED

Every raw retrieval result begins as LEAD. A model score alone is not evidence that a vulnerability exists.

Training procedure

Data construction

• 36,618 implemented Solidity functions from 99 projects.
• 208 training findings.
• 57 repository-disjoint validation findings.
• 63 unseen-protocol-family test findings.
• 27 separately sealed regression findings.
• Conservative grouping keeps versions, forks, upgrades, and related audits in the same protocol family.
• Unreported functions are treated as distractors, never certified-safe negatives.
• The corpus is not included in this repository.

The initial source index was reconstructed from a pinned FORGE-Curated snapshot. Source-level provenance and redistribution rights remain under review; see License and provenance.

Objective and optimization

A 149M-parameter LateOn-Code checkpoint also fit on the 4 GB GPU at 3.364 GiB, but lost the frozen validation bakeoff. The smaller 17M model was selected on evidence, not size or novelty.

Evaluation protocol

Three seeds were frozen before the unseen-family test was opened.

The primary evaluation ranks all implemented functions in each held-out repository rather than selecting among a small set of sampled candidates. Repository versions, forks, upgrades, and related audits are grouped into protocol families to reduce leakage. The reported test contains 63 findings from nine unseen families; a separate sealed regression set contains 27 findings.

Results

• Protocol-family-clustered 95% bootstrap interval for test ΔMRR: [0.0002, 0.2282].
• Exact family sign-flip test: p=0.230 because the test contains only nine independent families.
• All three seeds improved Recall@10 on both held-out sets.
• The predeclared +5 percentage-point improvement gate passed.
• The stronger 60% Recall@10 / 80% Recall@20 competitiveness gates failed.

The model passed the predeclared improvement gate. It failed the stronger competitiveness thresholds. The direction of improvement is repeatable across the three runs; the absolute recall remains insufficient for independent security review.

Structural-context ablation

The slice improved early ranking, especially Recall@5. It is a deterministic no-build approximation—not Slither and not a complete inter-contract graph.

Post-freeze external pilot

The fixed generic mechanism scan was also run on nine findings from four official ReEVMBench task repositories after the main result was frozen.

Five findings improved, two tied, and two worsened. No scored positive exactly matched training code. This is not an official ReEVMBench Detect score: the sample is tiny, gold was used only for post-hoc localization scoring, and the task repositories lacked the base_commit objects declared by their benchmark configs. Full predictions are preserved in reevm_external_summary.json.

Negative result: hard-negative collapse

One model-mined hard-negative continuation collapsed validation Recall@10 from 59.6% at the starting checkpoint to 47.4% after 25 steps, 26.3% after 50, and 19.3% after 100. The selector retained step 1.

This result suggests that automatically treating highly ranked, unlabeled functions as negatives can suppress useful evidence. Unreported code is therefore represented as a distractor rather than certified-safe supervision.

Intended use

Supported research uses:

• Prioritize functions and connected code neighborhoods for an authorized Solidity review.
• Supply a shortlist to a human auditor or a second-stage reasoning model.
• Compare retrieval behavior across general vulnerability mechanisms.
• Study repository-aware, token-level code retrieval under small-GPU constraints.

Out-of-scope uses:

• Certifying a contract as safe.
• Autonomous vulnerability discovery.
• Severity classification.
• Calibrated exploit probability.
• Running tests against systems, contracts, networks, or funds without authorization.

Limitations

• On the locked test, 82% of known positives remained outside the first ten results.
• Generic mechanisms can converge on the same suspicious function.
• Static slices omit build-resolved semantics, dynamic dispatch, and deeper cross-contract behavior.
• Upstream file/function mappings remain subject to manual verification.
• Second-stage hypothesis text can be plausible but wrong; executable evidence remains authoritative.
• The external pilot is too small for a benchmark or SOTA claim.

Future work

Future checkpoints will be considered competitive only if a new locked evaluation reaches:

• at least 60% Recall@10;
• at least 80% Recall@20;
• the same direction across three runs and unseen protocol families;
• reproducible Foundry tests for a meaningful share of the shortlisted leads.

Failure cases are particularly useful. Open a Community discussion with the repository commit, risk query, expected function, and returned rank so that future experiments can target a concrete retrieval failure.

License and provenance

The base checkpoint is Apache-2.0. The EVM-Scout training examples were reconstructed from public security reports and source snapshots whose per-source terms are still being reviewed. The corpus is not distributed here.

This is a public research preview while the derived-weight and per-source provenance review remains open. Public availability is not a representation that the fine-tuning corpus is redistributable, and the corpus is not included. No public or commercial redistribution license is granted for the fine-tuned weights by this preview. See LICENSE.md.

Reproducibility and evidence

• locked_summary.json — complete locked predictions and statistics.
• reevm_external_summary.json — post-freeze external pilot, task HEADs, hashes, and overlap checks.
• smoke-results.json — pinned Foundry, Slither, Echidna, and Halmos smoke evidence.
• security_mechanisms.json — fixed 12-query mechanism bank.
• scan_multivector_repo.py — repository scanner.
• solidity_graph.py — deterministic function and security-slice extractor.
• solidity_parser.py — dependency-free conservative Solidity function parser.

Acknowledgements

Built on lightonai/LateOn-Code-edge, Sentence Transformers MultiVectorEncoder, and the broader ColBERT/late-interaction ecosystem. Historical system experiments used google/gemma-4-E4B-it as a separate frozen reviewer; it is not part of this checkpoint.

Citation

@misc{espejel2026lateonevmscout,
  title        = {LateOn-Code-edge EVMScout 17M: Repository-Wide Solidity Security Retrieval on a 4GB GPU},
  author       = {Omar Espejel},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/starknet-ai/LateOn-Code-edge-EVMScout-17M}
}