Pyrrho v2 nano g1
Pyrrho is a CPU-runnable co-processor for retrieval-augmented generation. It reads a question before retrieval to suggest what evidence to seek, then reads the question with retrieved passages to assess whether those passages support an answer. A surrounding RAG runtime decides whether to answer, retrieve again, or surface a conflict. The broader project is described in the Pyrrho repository.
What this checkpoint does
| Pass | Input | Signals |
|---|---|---|
| Plan | Question | Retrieval intent and useful evidence types |
| Assess | Question and source passages | Evidence verdict, failure reason, retrieval intent, and evidence types |
The evidence verdict is SUFFICIENT, DISPUTED, or INSUFFICIENT. These are
predictions about the supplied passages. The model does not retrieve sources,
generate answers or citations, check external facts, or prove that a corpus has
been searched completely.
Input format
The two passes use the same encoder with different input prefixes:
[PYRRHO_PRE]
Question: <user question>
[PYRRHO_POST]
Question: <user question>
Sources:
[1] <retrieved passage>
[2] <retrieved passage>
Inputs are truncated at 2,048 tokens by the published inference setup. For the PRE pass, use only the retrieval-intent and evidence-kind heads. For the POST pass, use all four heads.
Output contract
The checkpoint produces 18 logits in four heads, not one 18-class distribution. Labels below are in logit order.
| Slice | Head | Labels | Decode |
|---|---|---|---|
0:3 | evidence_verdict | INSUFFICIENT, DISPUTED, SUFFICIENT | Softmax |
3:8 | failure_mode | none, unresolved_conflict, missing_or_incomplete_evidence, wrong_scope_or_version, ambiguous_request | Softmax |
8:12 | retrieval_intents | needs_lookup, needs_temporal_resolution, needs_comparison_or_set, needs_broad_coverage | Independent sigmoid scores |
12:18 | evidence_kinds | needs_text, needs_table_or_record, needs_code_or_symbol, needs_config_or_setting, needs_log_or_run_result, needs_document_layout | Independent sigmoid scores |
A generic Hugging Face text-classification pipeline does not decode these
heads correctly. A downstream runtime must decode each slice and apply its own
thresholds and decision policy. Softmax and sigmoid scores are not calibrated
assurances that an answer is safe.
Running locally
The repository contains a Transformers checkpoint (model.safetensors), an
FP32 ONNX export (model.onnx), and a dynamic INT8 ONNX export
(model_quantized.onnx). All can run on CPU; a GPU or hosted LLM is not required
for inference. The ONNX exports use the same 18-logit contract. The INT8 export
has not been qualified as an interchangeable evidence gate.
An integration should choose its own thresholds and test the resulting decisions on the evidence it expects to encounter.
Evaluation
The tables below reproduce the reported measurements for this checkpoint.
They describe the original development run and Fitz-Sage integration checks;
their limitations are explained after the tables. The development run was
modernbert_base_v2_dual_from_g1_41358_active_20260704_seed42.
Post-retrieval development split
| Metric | Result |
|---|---|
| Overall score | 0.9471 |
| Evidence-verdict accuracy | 0.9703 |
| False-SUFFICIENT rate | 0.0484 |
| Failure-mode accuracy | 0.9567 |
| Retrieval-intent exact match | 0.8308 |
| Retrieval-intent macro F1 | 0.9277 |
| Evidence-kind exact match | 0.9809 |
| Evidence-kind macro F1 | 0.9950 |
Pre-retrieval development split
| Metric | Result |
|---|---|
| Retrieval-intent exact match | 0.8248 |
| Retrieval-intent macro F1 | 0.9266 |
| Evidence-kind exact match | 0.9637 |
| Evidence-kind macro F1 | 0.9873 |
Fitz-Sage integration checks
| Suite | Result |
|---|---|
| Balanced fixed-evidence governance suite | 120/120 |
| Live benchmark, total | 97/120 |
| Core | 19/20 |
| Holdout | 43/50 |
| Holdout2 | 35/50 |
How to read these results
The 41,358-row training snapshot includes a deterministic 5,000-row tranche with only 32 unique post-retrieval model inputs. The original split used row IDs, so identical inputs appeared in both training and evaluation. The fixed-evidence suite also shared wording with that tranche. The development and fixed-suite scores therefore reproduce the original experiment, but are not independent estimates of generalization. The live benchmark is an integration check, not a substitute for a separate test set.
Later calibration diagnostic
A separate 2,500-example synthetic calibration comparison used a 0.34 runtime threshold for this published checkpoint:
| Metric | Result |
|---|---|
| Runtime verdict accuracy | 0.5044 |
| Retrieval-intent exact match | 0.1488 |
| Evidence-kind exact match | 0.2156 |
| False-SUFFICIENT 95% Wilson upper bound | 19.15% |
This calibration set is also synthetic and does not establish performance on real retrieval traffic.
Training and limitations
| Item | Detail |
|---|---|
| Base encoder | answerdotai/ModernBERT-base |
| Training snapshot | yafitzdev/pyrrho-v2 at revision 7224572 |
| Language | English |
| Training data | 41,358 synthetic RAG questions and evidence packs |
The model judges only the evidence it receives. Synthetic examples and benchmark-like inputs do not establish behavior on live retrieval traffic, other languages, unseen document formats, or high-stakes decisions. Use independent evaluation before relying on its verdict to stop retrieval.