Source model card: Falconsai/proof_v2 @ main, carried verbatim below. Its licence is the repository's. The Model Surgeon record follows it.

Falconsai/proof_v2

View in Model Surgeon

An intentional decision model for the edge. A ModernBERT-base encoder carries a zero-shot listwise generalist and three distilled specialist heads (support intents, code language and reasoning), each behind its own step-function gate. You supply a state, a question and a closed set of options; it returns a calibrated probability for each option and names the head that answered.

falconsproof v2.0 is the successor to Falconsai/proof_V_1. It swaps DistilBERT for ModernBERT, adds code and reasoning specialists, and trains on a code-heavy mix of 19 tasks (73.4% of training decisions are code). It powers the IntFalcon Decision Lab.

Read this first: the gates in this checkpoint are closed. The validation search found no gate setting that beat the generalist, so every gate kept its closed value (τ = κ = μ = 1.01). Every answer in this checkpoint comes from the generalist, and all the test results below are the generalist's. The specialist heads are trained, calibrated and available for inspection (§5.4), and on some tasks one alone is stronger. The intent specialist, for example, scores 99.2% on Bitext against the generalist's 95.8%. Re-tuning the gates (§13) is the most direct improvement available.

Status of the numbers. Every figure here comes from this checkpoint's falconsproof_report.json: one training run with the small preset (up to 500 training decisions per source and task, 1 epoch per stage, seed 42), trained on an NVIDIA GeForce RTX 5090 Laptop GPU in bfloat16.

Table of contents

1. Model summary
2. Results at a glance
3. Model details
4. Intended uses and out-of-scope uses
5. How to get started
6. Architecture
7. Training data
8. Training procedure
9. Calibration, fidelity and gates
10. Evaluation
11. Comparison with v1
12. Bias, risks and limitations
13. Recommendations
14. Environmental impact
15. Technical specifications
16. Files in this repository
17. Versioning
18. Citation and references
19. Model card authors and contact

1. Model summary

The model has no generative component. It can only rank the options you give it.

2. Results at a glance

Test accuracy on 5,096 decisions from 23 sources and tasks, 4 of them held out:

Calibration is excellent. Overall expected calibration error (ECE) is 0.008, so a stated confidence of 80% means about 80% accuracy on data like this test set. Answers are also independent of option order: accuracy after reshuffling the options is identical on every source.

3. Model details

4. Intended uses and out-of-scope uses

Intended

Out of scope

• Arithmetic and multi-step reasoning. GSM8K is at chance (27.5%), and science and commonsense multiple choice are weak.
• Security or correctness gating of code. Vulnerability detection (54.2%), clone detection (38.3%) and spotting single-token bugs (47.5%) are near or below chance. Don't use it as a code reviewer or a security scanner.
• Open-ended questions. It always picks one of your options; add "none of these" explicitly if that's a possible answer.
• Long inputs. Anything beyond 384 tokens (state and option together) is truncated.
• Non-English text, and high-stakes decisions without human oversight (medical, legal, financial, employment, safety-critical).

5. How to get started

5.1 Install

pip install torch "transformers>=4.48,<5" safetensors huggingface_hub numpy

ModernBERT needs transformers 4.48 or newer. This checkpoint was saved with transformers 4.57.1.

5.2 Load

import importlib.util
from huggingface_hub import snapshot_download

path = snapshot_download("Falconsai/proof_v2")
spec = importlib.util.spec_from_file_location("falconsproof_modeling", f"{path}/falconsproof_modeling.py")
fpm = importlib.util.module_from_spec(spec); spec.loader.exec_module(fpm)

model, tokenizer = fpm.FalconsProof.load(path)          # cuda if available, else cpu
print(model.config["version"], [s["name"] for s in model.specialists])

5.3 Decide (with question fanout)

result = fpm.decide(model, tokenizer,
    state="def mean(xs):\n    return sum(xs) / len(xs)",
    questions=[
        {"question": "Which programming language is this?", "options": ["Python", "Ruby", "Go"]},
        {"question": "What does this function do?",
          "options": ["Computes the average of a list", "Sorts a list", "Reverses a string"]},
    ])
for r in result["results"]:
    print(r["question"], "->", r["choice"], r["probs"], "via", r["route"])
print(result["forward_passes"], "forward passes")      # 2, however many questions

Each result carries choice, probs, route ("generalist" or a specialist's name), the generalist's own distribution, and one gates row per specialist with conf/tau, affinity/kappa, min_map/map_min, collision and passed.

5.4 Read each head directly

analyze() returns every head's view, including the specialists that the closed gates keep from answering:

out, meta = fpm.analyze(model, tokenizer, [{
    "state": "I forgot my password and can't sign in.",
    "question": "Which team should handle this message?",
    "options": ["recover password", "shipping", "invoicing"]}])
o = out[0]
print("answered by:", o["route"], "| generalist:", o["gen_probs"].round(3))
for name, view in o["views"].items():
    print(f"{name:10s} choice={view['choice']} conf={view['conf']:.3f} affinity={view['affinity']:.3f}")

5.5 Open a specialist's gate (optional)

Gates are plain numbers in model.config. To let the intent specialist answer confidently mapped support questions:

spec = next(s for s in model.config["specialists"] if s["name"] == "intent")
spec["gate"] = {"tau": 0.90, "kappa": 0.50, "map_min": 0.70}      # choose from your own validation data

Validate any hand-set gate on labelled data before relying on it. §13 describes the proper re-tuning.

6. Architecture

                         ┌─────────────── ModernBERT-base encoder (shared) ───────────────┐
 state, option texts ────┤─► mean-pool ─► intent head   (15 labels)  ◄ Falconsai/intent_classification
                         │             └► code head     (6 labels)   ◄ huggingface/CodeBERTa-language-id
 (question | state,      │                                                                  
   option) pairs ────────┤─► mean-pool ─► reasoning head (contradiction/entailment/neutral) ◄ cross-encoder/nli-deberta-v3-small
                         │             └► option head   (1 logit per option): the generalist
                         └────────────────────────────────────────────────────────────────┘
   gate_k = H(conf_k − τ_k) · H(affinity_k − κ_k) · H(map_k − μ_k) · [no collisions]
   answer = the most confident specialist whose gate passes, otherwise the generalist

All heads are two-layer MLPs (768 → 768 → GELU → dropout → output) on mean-pooled encoder states.

7. Training data

Every example is a decision (state, question, options, answer, plus source, task, domain), with options shuffled. In total there are 18,795 training, 4,520 validation and 5,096 test decisions; 73.4% of the training decisions are code.

7.1 Sources and tasks

Held-out sources were never used for training, calibration or gate tuning. Check each dataset's card for its license before you redistribute derived data.

7.2 Decisions per source and task

Tasks built from the six-language corpus are capped per language, which is why they have 3,000 training decisions against 500 for other tasks.

7.3 Bitext → Falconsai label mapping

Each Bitext intent is mapped to the intent teacher's majority label if at least 70% of 64 examples agree. All 27 Bitext intents passed, covering 14 of the 15 labels; appointment has no Bitext equivalent.

7.4 Distillation sets (Stage A)

Each specialist learns from 2,500 teacher-labelled inputs, with 300 more held back to measure fidelity. About 15–25% of each set is deliberately outside the specialist's domain, so the specialist also learns how its teacher behaves off-domain, which is exactly what the gates rely on.

8. Training procedure

9. Calibration, fidelity and gates

9.1 Temperatures and specialist fidelity

Fidelity is argmax agreement between a specialist and its teacher on held-back distillation inputs.

• The specialists were over-confident (every temperature is well above 1), the reasoning head most of all (T ≈ 5.5). Calibration softens them so that one threshold means the same across heads.
• Fidelity rose in stage B, because the anchor kept distilling while the encoder improved. The intent and code heads match their teachers closely.
• The reasoning head only partly learned its teacher (70%). One epoch of the small preset wasn't enough to distil a DeBERTa NLI model, which helps explain why its gate stayed closed.

9.2 Gates

The search started with every gate closed and kept a setting only if it improved domain-balanced validation accuracy. It ran from 0.8526 to 0.8526: no change, so every gate stayed closed.

The test data suggests why. Where a specialist is strong, the generalist is nearly as strong (Bitext 99.2% against 95.8%; SNLI must-be-true 94.2% against 90.8%). The domain-balanced objective weighs support as one domain among four, so opening a gate either gains too little or also lets the specialist answer questions it shouldn't. The code specialist (89.9% on language ID) is outright weaker than the generalist (99.7%).

10. Evaluation

10.1 Metrics

10.2 Results per source and task

10.3 Key findings

1. Code understanding is the strongest area. Language ID (99.7%), task → solution (99.2%), code ↔ description (about 96–97%) and function naming (90.1%) are all far above chance, and the held-out HumanEval completion task reaches 57.5% against a chance level of about 33%.
2. Fine-grained code judgement is not learned. Clone detection (38.3%, below chance), bug spotting (47.5%) and Devign (54.2%) show the model reads what code is about, not whether it is correct.
3. Intent generalisation is strong. Held-out Banking77 scores 88.3% and CLINC150 85.0%, well above chance (about 32%), with a label space never seen in training.
4. Reasoning is mixed. Entailment-style tasks work (SNLI about 90%, SciQ 69.2%), but arithmetic (GSM8K 27.5%), OpenBookQA and ARC-Challenge are at or near chance.
5. Specialists outperform the generalist in places, but the closed gates stop them answering: intent on Bitext (+3.3 pts), reasoning on SNLI must-be-true (+3.3 pts). The reasoning head also shows why gates matter: asked "which statement contradicts the text", it picks the entailed statement (3.3%).
6. Calibration is reliable. ECE is 0.008 overall and at most 0.200 on any single source.

11. Comparison with v1

The same sources appear in both test sets, but they are different random samples (v1 used 300 decisions per source with 2–6 options, v2 used 120 with 2–5), so treat the differences as indicative.

v2 roughly doubles accuracy on unfamiliar intent taxonomies (CLINC150, Banking77) and improves multiple choice. v1 remains slightly better on in-domain Bitext routing, where its specialist gate was open. v2 also adds the whole code domain, which v1 didn't cover.

12. Bias, risks and limitations

• Gates closed. All answers come from the generalist. The specialists add no accuracy in this checkpoint until their gates are re-tuned.
• Light training. The small preset (1 epoch per stage, at most 500 decisions per task) and a reasoning head at 70% teacher fidelity leave room for improvement.
• Closed world. The model always picks one of your options.
• Truncation. 384 tokens total; long code or documents lose content.
• Code correctness. It can't tell correct code from subtly broken code (bug spotting 47.5%). Don't use it to approve code.
• Distribution-dependent calibration. ECE was measured on this test mix; re-check it on your own traffic.
• Data provenance. Training data is English and mostly templated, crowd-sourced or scraped from public code. Biases in the source datasets, the teachers and ModernBERT's pre-training can carry into decisions.
• Automation risk. Automatic routing can systematically misroute users whose phrasing differs from the training data (dialects, non-native speakers, assistive phrasing). Monitor misroutes by group where possible.

13. Recommendations

• Re-tune the gates. Run the gate search on a larger validation set, or on labelled examples of your own traffic, with domain weights that reflect your use. The intent gate is the obvious first candidate. The gates are plain values in falconsproof_config.json.
• Retrain with the standard preset (up to 2,500 decisions per task, 2 stage-B epochs, 512 tokens) to strengthen the reasoning head and the weak code-judgement tasks.
• Use deferral. Act on high-confidence answers and send the rest to a human or an LLM; the low ECE makes this dependable.
• Evaluate on your own data (50–500 labelled decisions) before deploying, especially for any code or reasoning use.
• Log the route, confidence and model revision with every decision.

14. Environmental impact

Inference needs no GPU: two forward passes of a 151M-parameter encoder per call.

15. Technical specifications

16. Files in this repository

17. Versioning

Changelog. v2.0.0: first release, trained with the small preset (seed 42).

18. Citation and references

@misc{falconsai_proof_v2_2026,
  title        = {falconsproof v2.0: distilled specialists and a listwise generalist behind step-function gates},
  author       = {{Falconsai}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Falconsai/proof_v2}},
  note         = {ModernBERT-base backbone; successor to Falconsai/proof_V_1}
}

Methods. Warner et al. (2024), Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder (ModernBERT). Hinton, Vinyals and Dean (2015), Distilling the Knowledge in a Neural Network. Li and Hoiem (2016), Learning without Forgetting. Nogueira and Cho (2019), Passage Re-ranking with BERT. Cao et al. (2007), Learning to Rank: From Pairwise Approach to Listwise Approach. Guo et al. (2017), On Calibration of Modern Neural Networks. Geifman and El-Yaniv (2017), Selective Classification for Deep Neural Networks.

Data. Lu et al. (2021), CodeXGLUE. Austin et al. (2021), Program Synthesis with Large Language Models (MBPP). Chen et al. (2021), Evaluating Large Language Models Trained on Code (HumanEval). Larson et al. (2019), CLINC150. Casanueva et al. (2020), Banking77. Talmor et al. (2019), CommonsenseQA. Mihaylov et al. (2018), OpenBookQA. Welbl et al. (2017), SciQ. Clark et al. (2019), BoolQ. Cobbe et al. (2021), GSM8K. Bowman et al. (2015), SNLI. Williams et al. (2018), MultiNLI. Clark et al. (2018), ARC. Bitext, Customer Support LLM Chatbot Training Dataset.

19. Model card authors and contact

Written by the Falconsai team. Please report issues, misroutes or evaluation results through the Community tab of this repository.

This card is generated from the surgical record itself; the package's lineage.intoto.jsonl is the signed source of truth (verify it free at the Surgeon's public verifier or with the bundled verify_attestation.py).

Architecture

• Identification: NLP · Small Language Model (SLM) (62% confidence)
• Source format: safetensors · Intended task: not declared
• config.json: the repo's config.json, edited
• Source license: apache-2.0
• Lineage chain: 1 surgery (no prior attestation reachable) · Falconsai/proof_v2
• Post-surgery totals: 149,014,272 parameters · 134 tensors
• Compute estimate: 14.118027 GFLOPs (comparison metric, not a measurement)

Provenance & operations

• Parents: Falconsai/proof_v2/model.safetensors
• Operations performed: load×1
• Weight merges recorded: 0
• Quantized tensors (F32→F16): 0

Surgery Log (ordered)

1. load — hub:Falconsai/proof_v2/model.safetensors (596.1 MB, safetensors)

Validation

• Tissue imaging: not run
• Structural integrity is testable offline via the packaged load_and_test.py.

Compliance note

The signed attestation + this card together document model composition, modification history, and validation evidence — the record structure technical-documentation obligations (e.g. EU AI Act Annex IV) ask for. This is evidence, not legal advice.

Operated with Model Surgeon — verify this package at https://surgeon.falcons.ai/verify © 2026 FALCONS.AI — Model Surgeon record format. The model weights remain their owner's.