/ Модель

Характеристики

Контекст
512 токенов
Лицензия
Apache 2.0
Архитектура
falconsproof · dense
Базовая модель
intent_classification
Библиотека
transformers
Опубликована
24 сен 2026

proof_V_1

Классификация текстаТекст → label3 профессии

Англоязычная модель Falconsai на базе DistilBERT выбирает и оценивает варианты для маршрутизации обращений в службу поддержки.

185скачиваний за 30 дней1лайквеса обновлены24 сен 2026

Бенчмарки

Как модель справилась с задачами пользователей ailist и что о ней публикуют открытые лидерборды

Публичные оценки

Из открытых лидербордов. Сравнивайте результаты одного теста при одинаковых условиях — отсутствие оценки не значит, что модель хуже.

В подключённых источниках результатов для этой модели пока нет.

Что умеет модель

Описание составлено по карточке модели и метаданным Hugging Face

falconsproof v1 — модель Falconsai для выбора и ранжирования вариантов ответа по заданному текстовому состоянию, вопросу и списку из 2–20 вариантов. Она возвращает калиброванную вероятность для каждого варианта и не генерирует текст. Модель построена на DistilBERT с 6 слоями, 768 скрытыми параметрами и 12 головами внимания; содержит около 67,6 млн параметров. Контекст модели — 512 токенов по метаданным, однако в описанных сценариях вход обрезается после 128 токенов.

Основное применение — маршрутизация англоязычных обращений в службу поддержки по закрытому набору намерений. Внутри домена Bitext модель показала точность 97,3%, но на незнакомых банковских намерениях качество и калибровка заметно хуже. Для открытых вопросов, генерации текста, длинных документов, многошаговых рассуждений и языков кроме английского модель не предназначена; для важных решений требуется человеческий контроль. Лицензия — Apache-2.0.

Сценарии применения

  • классификация намерений в обращениях клиентов
  • маршрутизация сообщений в службу поддержки по заданным категориям
  • выбор наиболее подходящей метки из закрытого списка вариантов
  • оценка уверенности в выборе категории обращения

Ограничения

Модель всегда выбирает один из предложенных вариантов, даже если правильного среди них нет; в описанных сценариях текст обрезается после 128 токенов. Качество и калибровка зависят от домена, а на задачах за пределами маршрутизации поддержки, включая научные и commonsense-вопросы, результаты близки к случайным.

Для каких профессий подходит3

Разделы каталога профессий, в задачах которых используется эта модель

Все профессии

Карточка модели

README от автора на Hugging Face, как он его опубликовал

Открыть на Hugging Face

Source model card: Falconsai/proof_V_1 @ main, carried verbatim below. Its licence is the repository's. The Model Surgeon record follows it.

falconsproof v1 (Falconsai/proof_V_1)

View in Model Surgeon

A specialist–generalist decision model on a single DistilBERT encoder. The original 15-intent customer-support classifier is kept intact; a listwise cross-encoder head accepts any options you define at request time; and a step-function gate, tuned on validation data, decides which of the two answers each question.

falconsproof v1 extends Falconsai/intent_classification, pinned at revision 630d0d4668170a2a64d8d80b04d9844415bd4367. It is the first version of the falconsproof line and the direct predecessor of falconsproof v2.0 (ModernBERT backbone, three specialists). This card describes v1 only.

Status of the numbers in this card. Every number comes from this checkpoint's config.json and falconsproof_report.json: one training run with the small preset (1,500 training decisions per source, 1 epoch, seed 42). The tables between <!-- … --> markers are generated by fill_model_card.py; the analysis around them is written by hand. Only the environmental-impact section (§14) is still to be recorded.


Table of contents

  1. Model summary
  2. Model details
  3. Intended uses
  4. Out-of-scope uses
  5. How to get started
  6. Architecture
  7. Training data
  8. Training procedure
  9. Calibration and gate parameters
  10. Evaluation
  11. Behavioural test suite
  12. Bias, risks and limitations
  13. Recommendations
  14. Environmental impact
  15. Technical specifications
  16. Files in this repository
  17. Versioning and relation to v2
  18. Glossary
  19. Citation and references
  20. Model card authors and contact

1. Model summary

TaskClosed-set decisions: given a state (text), a question, and 2–20 options, return a calibrated probability per option
BackboneDistilBERT (6 layers, 768 hidden, 12 heads), initialised from Falconsai/intent_classification
Parameters67,556,476 in 109 tensors, float32 (as counted by Model Surgeon): encoder ≈ 66.4M; intent head ≈ 0.6M; option head ≈ 0.6M
HeadsSpecialist: the original 15-intent head, frozen and preserved by distillation. Generalist: a listwise cross-encoder option head
RouterStep-function gate H(conf − τ) · H(coverage − κ) · H(map_conf − μ) · [no collisions], thresholds found by exhaustive search: τ = 0.875, κ = 0.20, μ = 0.50
CalibrationOne temperature per head, fitted by NLL on validation data: t_intent = 1.000, t_option = 0.709
Context128 tokens (state and pairs are truncated beyond this)
Inference costTwo encoder forward passes per call, however many questions are asked about the same state
LanguageEnglish
OutputProbabilities, the chosen option, which head answered (specialist / generalist), and the gate's inputs
Headline results (test)Support routing (Bitext): 97.3%, above either head alone (specialist 95.0%, generalist 84.3%). All sources: 45.3% (specialist only 43.5%, generalist only 43.2%). Held-out Banking77: 45.0%. Multiple-choice QA: at or near chance

The model has no generative component. It cannot produce text, only rank the options you give it.


2. Model details

FieldValue
Model namefalconsproof v1
RepositoryFalconsai/proof_V_1
Developed byFalconsai
Model typeEncoder-only decision model: classifier head plus cross-encoder ranking head with a thresholded router
Architecture classFalconsProofModel (subclass of DistilBertPreTrainedModel), model_type: "falconsproof"
Config classFalconsProofConfig (subclass of DistilBertConfig)
Fine-tuned fromFalconsai/intent_classification @ 630d0d4668170a2a64d8d80b04d9844415bd4367
LanguageEnglish
LicenseApache-2.0 for the model weights and code (verify against the base model's license before redistribution; see §7.4 for data licenses)
Custom codeYes: falconsproof_modeling.py ships with the weights. It is loaded with importlib, not with trust_remote_code
Training notebookfalconsproof.ipynb (v1)
Intent labels (15)cancellation, ordering, shipping, invoicing, billing and payment, returns and refunds, complaints and feedback, speak to person, edit account, delete account, delivery information, subscription, recover password, registration problems, appointment

3. Intended uses

3.1 Direct use

falconsproof v1 is intended for fast, local, closed-menu decisions over short English text, especially where one part of the traffic is customer-support intent routing and the rest is ad hoc.

UseExample
Support routingState: an inbound message. Options: the 15 intents, or your own phrasings of them
Runtime-defined triageState: a ticket. Options: your team names, written in plain language, changed at any time. Expect accuracy around 45% on unfamiliar taxonomies (CLINC150, Banking77), so use deferral (§13)
Short multiple-choice questionsNot recommended. Test accuracy on CommonsenseQA, OpenBookQA and ARC-Easy is at or near chance (§10.3)
Question fanoutSeveral questions (team, urgency, next step) about one message in one call
Selective automationAct automatically on specialist-routed or high-confidence answers, and send the rest to a human

3.2 Downstream use

The model can serve as a routing or verification component inside larger systems: an agent's tool-selection step, a pre-filter in front of an LLM, or a triage stage in a support pipeline. Its route output and calibrated confidence make a natural deferral signal (see §13).

3.3 Who should use it

Engineers who need sub-100 ms, deterministic, locally hosted decisions with interpretable routing, and who can evaluate the model on a sample of their own traffic before deploying.


4. Out-of-scope uses

falconsproof v1 is not suitable for:

  • Generating text, answers or explanations. It can only choose among options you supply.
  • Open-ended questions where the right answer might not be among the options. It always picks one; it has no built-in "none of the above". Add such an option explicitly if you need one.
  • Long documents. Inputs beyond 128 tokens are truncated; information past the cut-off is invisible to the model.
  • Code, math, commonsense or science questions, or multi-step reasoning. The generalist scores at or near chance on multiple-choice QA (§10.3). falconsproof v2.0 adds code and reasoning specialists.
  • Languages other than English.
  • High-stakes decisions without human oversight: medical, legal, financial, employment, credit, safety-critical or law-enforcement contexts.
  • Decisions about people based on protected characteristics, or any profiling use.
  • Security classification (vulnerability, abuse or fraud detection) without a dedicated, evaluated model.

5. How to get started

5.1 Install

pip install torch "transformers>=4.44,<5" safetensors huggingface_hub numpy

5.2 Load

The model uses a custom class, which is stored in the repository and loaded from the downloaded snapshot. The repository holds model_edited.safetensors rather than model.safetensors and has no tokenizer files, so this loader handles both: it reads the weights under either name and falls back to the base model's tokenizer, which is the one v1 was trained with.

import importlib.util, json
import numpy as np, torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import load_file
from transformers import AutoTokenizer

REPO = "Falconsai/proof_V_1"
BASE, BASE_REV = "Falconsai/intent_classification", "630d0d4668170a2a64d8d80b04d9844415bd4367"
files = list_repo_files(REPO)

# Model classes from the repo's modeling file
spec = importlib.util.spec_from_file_location("falconsproof_modeling", hf_hub_download(REPO, "falconsproof_modeling.py"))
fpm = importlib.util.module_from_spec(spec); spec.loader.exec_module(fpm)

# Config: use the repo's if complete, else the base DistilBERT config + falconsproof calibration and gate
raw = json.load(open(hf_hub_download(REPO, "config.json"), encoding="utf-8"))
report = (json.load(open(hf_hub_download(REPO, "falconsproof_report.json"), encoding="utf-8"))
          if "falconsproof_report.json" in files else {})
config = (fpm.FalconsProofConfig.from_dict(raw) if {"dim", "n_layers", "id2label"} <= raw.keys()
          else fpm.FalconsProofConfig.from_pretrained(BASE, revision=BASE_REV))
cal, gate = report.get("calibration", {}), report.get("gate", {})
for key, fallback in [("t_intent", cal.get("t_intent")), ("t_option", cal.get("t_option")), ("gate_tau", gate.get("tau")),
                      ("gate_kappa", gate.get("kappa")), ("gate_map_min", gate.get("map_min")),
                      ("option_template", None), ("max_len", None)]:
    value = raw.get(key, fallback)
    if value is not None:
        setattr(config, key, value)

# Weights: accept model_edited.safetensors and strip any key prefix the editing tool added
weights_file = next((f for f in ("model.safetensors", "model_edited.safetensors") if f in files), None) \
    or next(f for f in files if f.endswith(".safetensors"))
state = load_file(hf_hub_download(REPO, weights_file))
model = fpm.FalconsProofModel(config)
expected = set(model.state_dict())
strip = lambda k, p: k[len(p):] if k.startswith(p) else k
prefix = max(["", "model.", "module.", "falconsproof."], key=lambda p: len(expected & {strip(k, p) for k in state}))
missing, unexpected = model.load_state_dict({strip(k, prefix): v for k, v in state.items()}, strict=False)
missing = [k for k in missing if not k.endswith("position_ids")]
assert not missing, f"weights missing from {weights_file} (they would be random): {missing[:10]}"
if unexpected:
    print("ignored unexpected tensors:", unexpected[:10])
model.eval()                                     # add .to("cuda") for a GPU

# Tokenizer: the repo's if present, else the base model's (identical to what v1 was trained with)
try:
    tokenizer = AutoTokenizer.from_pretrained(REPO)
except Exception:
    tokenizer = AutoTokenizer.from_pretrained(BASE, revision=BASE_REV)
    print("tokenizer: not in the repo, using the base model's")

cfg = model.config
LABELS = [cfg.id2label[i] for i in range(cfg.num_labels)]
print(f"loaded {weights_file}: {len(LABELS)} labels | tau={cfg.gate_tau:.3f} kappa={cfg.gate_kappa:.3f} "
      f"mu={cfg.gate_map_min:.2f} t_intent={cfg.t_intent:.3f} t_option={cfg.t_option:.3f}")
# Expected for this checkpoint: tau=0.875 kappa=0.200 mu=0.50 t_intent=1.000 t_option=0.709

5.3 Decide (with question fanout)

In v1, the routing logic lives in the training notebook rather than in falconsproof_modeling.py. This self-contained function reproduces it exactly:

def softmax(x, t=1.0):
    x = np.asarray(x, dtype=np.float64) / max(float(t), 1e-6)
    e = np.exp(x - x.max())
    return e / e.sum()


@torch.no_grad()
def decide(state, questions):
    """questions: [{"question": str, "options": [str, ...]}, ...]  ->  one result per question (2 forward passes)."""
    dev = model.device
    # Pass 1: intent logits for the state and every unique option text (specialist).
    singles = list(dict.fromkeys([state] + [o for q in questions for o in q["options"]]))
    enc = tokenizer(singles, truncation=True, max_length=cfg.max_len, padding=True, return_tensors="pt")
    il = model.intent_logits(enc["input_ids"].to(dev), enc["attention_mask"].to(dev)).float().cpu().numpy()
    table = dict(zip(singles, il))
    # Pass 2: one relevance logit per (question | state, option) pair (generalist).
    firsts, seconds, spans = [], [], []
    for q in questions:
        first = cfg.option_template.format(question=q["question"], state=state)
        spans.append((len(firsts), len(firsts) + len(q["options"])))
        firsts += [first] * len(q["options"]); seconds += list(q["options"])
    enc2 = tokenizer(firsts, seconds, truncation="longest_first", max_length=cfg.max_len, padding=True, return_tensors="pt")
    scores = model.option_scores(enc2["input_ids"].to(dev), enc2["attention_mask"].to(dev)).float().cpu().numpy()

    results = []
    for q, (a, b) in zip(questions, spans):
        generalist = softmax(scores[a:b], cfg.t_option)
        native = softmax(table[state], cfg.t_intent)
        maps, map_conf = [], []
        for o in q["options"]:
            p = softmax(table[o], cfg.t_intent)
            maps.append(int(p.argmax())); map_conf.append(float(p.max()))
        specialist = softmax(np.asarray(table[state])[maps], cfg.t_intent)
        coverage = float(native[sorted(set(maps))].sum())
        collision = len(set(maps)) < len(maps)
        use_specialist = (not collision and min(map_conf) >= cfg.gate_map_min
                          and coverage >= cfg.gate_kappa and specialist.max() >= cfg.gate_tau)
        probs = specialist if use_specialist else generalist
        results.append({
            "question": q["question"],
            "choice": q["options"][int(np.argmax(probs))],
            "confidence": float(probs.max()),
            "route": "specialist" if use_specialist else "generalist",
            "probs": {o: round(float(p), 4) for o, p in zip(q["options"], probs)},
            "gate": {"conf": float(specialist.max()), "coverage": coverage, "min_map_conf": min(map_conf),
                     "collision": collision, "mapped": [LABELS[m] for m in maps]},
        })
    return results


for r in decide(
    "I was charged twice for my March invoice, please reverse the duplicate payment.",
    [{"question": "Which team should handle this message?", "options": ["billing and payment", "shipping", "delete account"]},
     {"question": "How urgent is this?", "options": ["Low", "Medium", "High"]}],
):
    print(r["question"], "->", r["choice"], f"({r['confidence']:.0%}, {r['route']})")

Tips for writing options. Options that are the exact intent names, or close paraphrases of them, give the specialist a confident mapping, which lets its gate open. Options that don't correspond to any of the 15 intents are handled by the generalist. Keep options distinct: two options that map to the same intent create a collision, and the specialist then abstains.


6. Architecture

                    ┌──────────────────── shared DistilBERT encoder (from Falconsai) ─────────────────────┐
 state ─────────────┤─► [CLS] ─► intent head (15 labels, frozen) ─► specialist distribution over options  │──┐
 option texts ──────┤─► [CLS] ─► intent head ─► maps each option to an intent (+ mapping confidence)      │  │
 (question | state, │                                                                                      │  │
   option) pairs ───┤─► [CLS] ─► option head (MLP → 1 logit per pair) ─► generalist distribution          │──┤
                    └──────────────────────────────────────────────────────────────────────────────────────┘  │
                                                                                                              ▼
      step-function gate:  specialist  ⇔  H(conf − τ) · H(coverage − κ) · H(min_map_conf − μ) · [no collisions]

6.1 Shared encoder

This is DistilBERT, initialised from the Falconsai checkpoint, and both heads read its [CLS] vector. During training the embeddings and the first two transformer layers are frozen, while the upper four layers adapt.

6.2 Specialist path

The pre_classifier → ReLU → dropout → classifier stack keeps exactly the parameter names of DistilBertForSequenceClassification, so the Falconsai weights load into it unchanged. It is frozen throughout training. At inference it does three jobs:

  1. Classifies the state into the 15 intents.
  2. Maps each option to an intent by classifying the option's own text, which also yields a mapping confidence.
  3. Restricts the state's distribution to the mapped intents and renormalises it, which gives a probability per option.

It also reports coverage: the share of the state's full 15-way probability that falls on the intents behind your options.

6.3 Generalist path

Each option is encoded as a pair "{question} | {state}" [SEP] "{option}", and a two-layer MLP (768 → 768 → GELU → dropout 0.2 → 1) turns the pair's [CLS] vector into a relevance logit. A softmax over one question's options gives the distribution. Because the options are inputs rather than output classes, the label space is defined at request time.

6.4 Gate

StepPasses whenGuards against
H(conf − τ)The specialist's calibrated top probability is ≥ τUncertain in-domain calls
H(coverage − κ)At least κ of the specialist's belief falls on the offered intentsInputs outside the specialist's domain
H(min_map_conf − μ)Every option maps to an intent with confidence ≥ μOptions the specialist can't interpret
[no collisions]Every option maps to a different intentOptions the specialist can't tell apart

When all steps pass, the specialist answers; otherwise the generalist does. The thresholds are stored in config.json as gate_tau, gate_kappa and gate_map_min; the initial value 1.01 means closed. In this checkpoint the gate is open, with τ = 0.875, κ = 0.20 and μ = 0.50 (§9).

6.5 Why this design

DecisionReason
One shared encoderOne artifact to deploy, and the generalist starts from support-domain representations
Frozen head plus distillationFreezing the head alone doesn't stop encoder drift. The distillation (KD) term keeps the specialist's outputs faithful to the teacher (the Learning-without-Forgetting pattern)
Cross-encoder, listwiseJoint attention over question, state and option, trained to produce a distribution over the options
Hard gate, searched rather than learnedStep functions have no useful gradient, and only three thresholds need choosing. Every answer then comes from one named head with an auditable reason
Domain-balanced objectiveStops the larger domain from dictating the thresholds
Coverage as a signalDetects inputs the specialist believes are about something that wasn't offered, even when its restricted confidence is high

7. Training data

7.1 Decision format

Every example is converted to one schema:

{"state": "…", "question": "…", "options": ["…", "…"], "answer": 1,
 "source": "clinc150", "domain": "general", "heldout": false, "intent_label": null}

Options are shuffled for every example, so the model can't learn a position shortcut.

7.2 Sources

SourceHugging Face idRoleSplits usedHow decisions are built
Bitext customer supportbitext/Bitext-customer-support-llm-chatbot-training-datasetIn-domain (train, val, test)carved from trainBitext intents are mapped to Falconsai labels empirically (§7.3). Options are Falconsai labels, shown as label names or as natural paraphrases in equal measure
CLINC150 (plus)clinc/clinc_oosGeneral intents (train, val, test)train / validation / test150 unseen intent names (plus "out of scope") as runtime-defined options
CommonsenseQAtau/commonsense_qaGeneral reasoning (train, val, test)train (train and val carved) / validation → test5-way multiple choice
OpenBookQA (main)allenai/openbookqaGeneral reasoning (train, val, test)train / validation / test4-way multiple choice
Banking77PolyAI/banking77Held out (test only)testUnseen intent label space
ARC-Easyallenai/ai2_arc (ARC-Easy)Held out (test only)testUnseen multiple-choice source

For intent sources, each decision has 2–6 options: the gold intent plus random distractors from the same label space. If Bitext can't be downloaded, the notebook falls back to a small synthetic in-domain set built from label paraphrases, and says so in its output.

7.3 Empirical Bitext → Falconsai label mapping

The first 64 examples of each Bitext intent are classified by the frozen Falconsai teacher. A Bitext intent is mapped to the teacher's majority label if at least 70% of those examples agree, and dropped otherwise. The 64 mapping examples per intent are then excluded from the training, validation and test pools. The resulting mapping is stored in falconsproof_report.json under bitext_intent_map.

27 Bitext intents were mapped.

Bitext intentFalconsai label
cancel_ordercancellation
change_orderordering
change_shipping_addressshipping
check_cancellation_feecancellation
check_invoiceinvoicing
check_payment_methodsbilling and payment
check_refund_policyreturns and refunds
complaintcomplaints and feedback
contact_customer_servicespeak to person
contact_human_agentspeak to person
create_accountedit account
delete_accountdelete account
delivery_optionsdelivery information
delivery_perioddelivery information
edit_accountedit account
get_invoiceinvoicing
get_refundreturns and refunds
newsletter_subscriptionsubscription
payment_issuebilling and payment
place_orderordering
recover_passwordrecover password
registration_problemsregistration problems
reviewcomplaints and feedback
set_up_shipping_addressshipping
switch_accountedit account
track_orderordering
track_refundreturns and refunds

All 27 Bitext intents passed the 70% agreement threshold, so none were dropped. They cover 14 of the 15 Falconsai labels: Bitext has no appointment intent, so appointment is never tested in-domain. Some mappings reflect how the Falconsai classifier sees them rather than the Bitext names, for example create_account → edit account (not registration problems) and track_order → ordering (not delivery information).

7.4 Dataset licenses

Check each dataset's card before you redistribute derived data. At the time of writing, the listed licenses are:

DatasetLicense
Bitext customer supportCDLA-Sharing-1.0
CLINC150CC BY 3.0
CommonsenseQAMIT
OpenBookQAApache-2.0
Banking77CC BY 4.0
ARCCC BY-SA 4.0

7.5 Data volumes

Volumes are set by the training preset: at most per_source_train decisions per source for training, and per_source_eval for validation and for test.

PresetTrain per sourceVal / test per sourceEpochsSelected when
smoke250801Pipeline check on a CPU
small1,5003001auto on a CPU
standard4,0005002auto on a GPU

Actual counts per split and source for this checkpoint:

Sourcetrainvaltest
arc_easy00300
banking7700300
bitext1500300300
clinc1501500300300
commonsense_qa1485297296
openbookqa1498300300
Total598311971796

Totals: 5,983 training, 1,197 validation and 1,796 test decisions. The test split includes 600 held-out decisions (Banking77 and ARC-Easy). Multiple-choice sources lose a few rows to malformed items.

7.6 Known data caveats

  • Bitext overlap. The Falconsai training set is not public but appears closely related to Bitext. In-domain specialist accuracy on Bitext may therefore be optimistic, and Banking77 is the fairer test of intent generalisation.
  • Label-space mismatch. Bitext intents that don't map cleanly to one Falconsai label (below 70% purity) are excluded, so the in-domain set covers only the unambiguous part of the taxonomy.
  • Synthetic options. Paraphrased options come from a fixed phrase list, two phrasings per label, so real users' wording will vary more.

8. Training procedure

8.1 Initialisation and teacher check

The student is built with FalconsProofModel.from_pretrained("Falconsai/intent_classification", revision=…). A frozen copy of the original classifier is loaded as the teacher. Before training, the student's intent logits must match the teacher's to within 1e-4, and the notebook stops if they don't.

8.2 Objective

L = CE_listwise(option logits of each decision, answer)  +  λ · T² · KL( softmax(teacher/T) ‖ softmax(student/T) )

The listwise cross-entropy is computed over each decision's options, with padding masked at −1e4. The distillation term uses the states of the training batch as inputs, with λ = 1.0 and T = 2.0.

8.3 Frozen and trainable parameters

ComponentTrainable
EmbeddingsNo
Transformer layers 1–2No
Transformer layers 3–6Yes (lr 3e-5)
Intent head (pre_classifier, classifier)No
Option headYes (lr 5e-4)

8.4 Hyperparameters

HyperparameterValue
OptimiserAdamW, weight decay 0.01
Learning rate, encoder / option head3e-5 / 5e-4
ScheduleLinear, 6% warmup
Batch16 decisions (flattened to all their option pairs)
Gradient clipping1.0
Maximum sequence length128
Options per decision2–6 when sampled, or all choices for multiple-choice sources
Distillation λ / T1.0 / 2.0
Precisionbf16 autocast on CUDA when supported, fp32 otherwise
Seed42
Checkpoint selectionBest generalist validation accuracy across epochs

Monitored during training: listwise CE, distillation KL, generalist validation accuracy, and specialist agreement with the teacher on in-domain validation states.

8.5 Calibration

After training, one temperature per head is fitted by minimising NLL on the validation set, using L-BFGS over log T and clamping the result to [0.05, 20]. t_option is fitted on every validation decision; t_intent is fitted on the 15-way intent logits of in-domain validation states. Calibration changes confidence but never the argmax.

8.6 Gate optimisation (the step-function search)

For each validation decision, the search computes the specialist's calibrated view (confidence, coverage, minimum mapping confidence, collisions) and the generalist's calibrated choice. It then evaluates every combination of:

ThresholdGrid
τ (confidence)41 values, 1.00 → 0.00
κ (coverage)21 values, 1.00 → 0.00
μ (mapping confidence)0.9, 0.7, 0.5, 0.3, 0.0

That is 4,305 settings in all, vectorised in NumPy. The objective is domain-balanced accuracy, the mean of in-domain accuracy and general accuracy. Grids run from high to low, so ties go to the most conservative gate, the one that routes the fewest decisions to the specialist.


9. Calibration and gate parameters

These values are read from this checkpoint's config.json and falconsproof_report.json.

ParameterValue
Intent-head temperature t_intent1.000
Option-head temperature t_option0.709
Confidence step τ (gate_tau)0.875
Coverage step κ (gate_kappa)0.200
Mapping step μ (gate_map_min)0.50
Validation objective (domain-balanced accuracy)0.6784
Validation decisions routed to the specialist31.4%

Base model: Falconsai/intent_classification@630d0d4668170a2a64d8d80b04d9844415bd4367.

What these values mean in practice. The specialist answers a question only when four things hold. Its calibrated top probability is at least 87.5%. At least 20% of its belief falls on the intents behind the offered options. Every option maps to an intent with at least 50% confidence. And no two options share an intent. On validation data that happened for 31.4% of decisions, and the gated model reached 67.8% domain-balanced accuracy.

Temperatures. t_option = 0.709 is below 1, so the generalist's raw scores were under-confident and calibration sharpens them. t_intent is exactly 1.000, so the specialist's probabilities are the intent head's raw softmax. An exact 1.0 can also mean the fit never moved from its starting value. Since the specialist's ECE on Bitext is low (0.025, §10), this doesn't hurt in-domain, but it is worth re-checking when retraining.

Reading the values. A τ close to 1 means the specialist answers only when it is nearly certain. A κ close to 1 means it answers only when the options cover almost all of its belief. A gate value above 1 means that condition never passes, so the specialist is effectively disabled.


10. Evaluation

10.1 Protocol

The test split contains all in-domain and general sources, plus the held-out sources Banking77 and ARC-Easy. Held-out data was never used for training, calibration or gate tuning. Metrics are reported per source and for all sources together:

MetricDefinition
SpecialistAccuracy of the specialist path alone, ignoring the gate
GeneralistAccuracy of the generalist path alone
falconsproofAccuracy of the gated model, the one you deploy
BalancedMean per-class recall of the gated model, where the class is the position of the correct option
PerturbedAccuracy after the options are reshuffled (seed 49), which tests position robustness
ECEExpected calibration error of the final probabilities (10 equal-width bins)
RouteShare of decisions answered by the specialist

10.2 Results

SourcenSpecialistGeneralistfalconsproofBalancedPerturbedECERoute → specialist
bitext3000.9500.8430.9730.9750.9730.02587.7%
clinc1503000.4530.4500.4630.4140.4630.11816.3%
commonsense_qa2960.2260.2500.2500.2490.2500.0685.7%
openbookqa3000.2530.2800.2670.2670.2670.1106.7%
arc_easy (held out)3000.2430.3170.3130.4490.3130.0895.0%
banking77 (held out)3000.4830.4470.4500.3710.4500.22118.3%
All sources17960.4350.4320.4530.4620.4530.08723.3%

Preset small, seed 42, max length 128, 1 epoch(s). Accuracies are fractions of 1.

10.3 Key findings

SourceChance*Best single headfalconsproofGain from gatingSpecialist route
Bitext (in-domain)≈ 0.290.950 (specialist)0.973+2.3 pts87.7%
CLINC150≈ 0.290.453 (specialist)0.463+1.0 pts16.3%
Banking77 (held out)≈ 0.290.483 (specialist)0.450−3.3 pts18.3%
CommonsenseQA0.200.250 (generalist)0.2500.0 pts5.7%
OpenBookQA0.250.280 (generalist)0.267−1.3 pts6.7%
ARC-Easy (held out)≈ 0.250.317 (generalist)0.313−0.3 pts5.0%
All sources0.435 (specialist)0.453+1.8 pts23.3%

*Chance is the expected accuracy of a random pick. Intent decisions have 2–6 options chosen uniformly at random (mean of 1/k ≈ 0.29); CommonsenseQA has 5 options and OpenBookQA 4.

  1. The gate does its job in the home domain. On Bitext the gated model (97.3%) beats the specialist alone (95.0%) and the generalist alone (84.3%). The gate sends the 87.7% of decisions the specialist is sure about to the specialist, and lets the generalist handle the rest.
  2. Combining helps overall. Across all sources the gated model is 1.8 points above the best single head.
  3. It under-routes on unfamiliar banking intents. On held-out Banking77 the specialist alone would have scored 48.3%, but the gate sent it only 18.3% of decisions, giving 45.0%. Re-tuning the gate on banking-style traffic should recover this (see the developer guide's tune_gate).
  4. The generalist doesn't reason. On CommonsenseQA, OpenBookQA and ARC-Easy it scores at or barely above chance. The small preset (1 epoch, 1,500 decisions per source) and a 67M-parameter encoder were not enough to learn multiple-choice QA. Treat any use outside support routing as unproven.
  5. No position bias. Perturbed equals falconsproof on every source. This is expected by design: each option is scored independently, so reordering the options can't change the choice.
  6. Calibration is good in-domain and poor on Banking77. ECE is 0.025 on Bitext but 0.221 on Banking77, where confidence overstates accuracy. Don't use a single confidence threshold across domains without checking it.

10.4 How to read the results

  • Bitext. Compare specialist with falconsproof. The gate should keep most of the specialist's in-domain accuracy.
  • CLINC150, CommonsenseQA and OpenBookQA. falconsproof should track the generalist, and route should be low.
  • Banking77 and ARC-Easy (held out). These are the honest zero-shot numbers.
  • Perturbed close to falconsproof means the model reads the options rather than their positions.
  • A low ECE means that confidence thresholds, such as a deferral cut-off, behave as expected.

10.5 Reproducing the evaluation

Run falconsproof.ipynb top to bottom with preset = "small" and seed 42. The notebook writes the table above to falconsproof_report.json. Results vary with the preset, the hardware (bf16 versus fp32), and dataset versions on the Hub.


11. Behavioural test suite

The training notebook runs these checks on the model after reloading it from disk. The notebook's printed output records whether each one passed for a given training run:

#CheckPass criterion
1Probabilities are validEach question's probabilities sum to 1 (±1e-6)
2DeterminismTwo identical calls return identical probabilities
3Save/load fidelityOption and intent logits match to <1e-4; temperatures, gate and template round-trip; test metrics identical after reload
4Fanout consistencyAsking questions together equals asking them separately (±1e-3)
5Order robustnessThe chosen option text is unchanged after shuffling, on ≥ 90% of 200 test decisions
6In-domain routing"I forgot my password and can't sign in" with intent-name options routes to the specialist
7Off-domain routingA deployment question ("canary shows a 3% error increase") routes to the generalist
8Specialist preservedThe intent head agrees with the original Falconsai model on ≥ 95% of in-domain test states

A failing check is treated as a finding about the checkpoint, typically an undertrained preset, rather than as a crash.


12. Bias, risks and limitations

Technical limitations

  • Closed world. The model always picks one of the options, even when none is right.
  • 128-token window. Long messages are truncated, which can remove the deciding detail.
  • Small backbone, light training. A 67M-parameter DistilBERT trained with the small preset: multiple-choice accuracy on science and commonsense questions is at or near chance (25–31%).
  • Fixed specialist vocabulary. The specialist knows only 15 support intents. Everything else relies on the generalist.
  • Calibration is distribution-dependent. Temperatures were fitted on the validation mix. On held-out Banking77 the calibration error is 0.221 (versus 0.025 in-domain), so confidence can be badly overstated on unfamiliar traffic.
  • Mapping errors. The specialist maps options by classifying their text. Unusual phrasings can be mapped to the wrong intent with high confidence. The μ threshold and the collision check reduce this risk but don't remove it.

Sociotechnical risks

  • Language and dialect. The training data is English and mostly templated or crowd-sourced. Performance on dialects, code-switching, informal spelling or non-native phrasing is untested and likely lower.
  • Inherited bias. Biases in the base model's pre-training data, in Bitext's synthetic customer language, and in the QA datasets can carry over into the decisions.
  • Automation risk. Routing customers automatically can systematically misroute groups whose phrasing differs from the training data, for example people with disabilities who use assistive phrasing, or non-native speakers.
  • Over-trust in confidence. A calibrated confidence is an average property. It doesn't guarantee that any single decision is right.

13. Recommendations

  • Evaluate on your own traffic (50–500 labelled decisions) before deploying, and compare the results with §10.
  • Use a deferral policy. Act on specialist-routed answers, or on generalist answers above a confidence threshold chosen from your evaluation. Send everything else to a human.
  • Keep humans in the loop for irreversible or consequential actions.
  • Log the route, confidence and model revision with every decision, and review the misroutes regularly.
  • Monitor drift. Re-check calibration (ECE) whenever your traffic, product lines or channels change.
  • Retrain with the standard preset on a GPU (4,000 decisions per source, 2 epochs) before relying on the generalist, and re-tune the gate on your own traffic if it resembles Banking77 more than Bitext.
  • Prefer falconsproof v2.0 for new work: it has a longer context, a stronger generalist, and code and reasoning specialists.

14. Environmental impact

Not yet filled. Record the hardware, training time and region of your training run here. Estimate emissions with the ML CO2 Impact calculator.

Hardware—
Training time—
Cloud provider / region—
Estimated emissions—

This checkpoint was trained with the small preset (about 6k training decisions, 1 epoch), which the notebook's auto setting selects on a CPU. As a guide, v1 is small: the standard preset runs about 10k decisions for 2 epochs on a 67M-parameter model, which takes minutes to tens of minutes on a single modern GPU. Inference needs no GPU.


15. Technical specifications

ObjectiveListwise CE + KD (T = 2)
Pooling[CLS] token of the final layer
Pair format"{question} | {state}" [SEP] "{option}", truncated longest_first to 128 tokens
Inference passes2 per call: (1) state and unique option texts through the intent head; (2) all (question, option) pairs through the option head
SoftwarePython ≥ 3.9 (3.11 recommended), PyTorch ≥ 2.1, transformers ≥ 4.44 and < 5, safetensors, NumPy. This checkpoint's config was saved with transformers 4.57.6
Serialisationmodel_edited.safetensors (float32, also as model_edited.F32.gguf) plus config.json (model_type: falconsproof, architectures: ["FalconsProofModel"])
DeterminismBit-for-bit repeatable in eval mode with the same hardware and library versions

Config fields added on top of DistilBertConfig:

FieldMeaningInitialThis checkpoint
t_intentIntent-head temperature1.01.000
t_optionOption-head temperature1.00.709
gate_tauConfidence step τ1.01 (closed)0.875
gate_kappaCoverage step κ1.01 (closed)0.200
gate_map_minMapping-confidence step μ1.01 (closed)0.50
option_templateFirst-segment template for pairs"{question} | {state}"same
max_lenTokenizer truncation length128128

The other DistilBERT settings in config.json are unchanged from the base model: 6 layers, 12 heads, dim 768, hidden dim 3072, vocabulary 30,522, 512 positions, dropout 0.1, and classifier dropout 0.2.


16. Files in this repository

FileContents
config.jsonArchitecture, 15 labels, temperatures, gate thresholds
model_edited.safetensorsAll weights (float32, 270 MB): encoder, intent head, option head
model_edited.F32.ggufThe same weights in GGUF format
falconsproof_modeling.pyFalconsProofConfig and FalconsProofModel
falconsproof_report.jsonBase-model revision, labels, calibration, gate search, Bitext mapping, training config, data counts, test results
lineage.intoto.jsonl, manifest.json, surgery_log.jsonModel Surgeon lineage attestation, manifest and operation log
falconsai_surgeon_package_*.zipThe complete Model Surgeon package
README.mdThis model card

The repository has no model.safetensors and no tokenizer files, so plain from_pretrained("Falconsai/proof_V_1") fails. The loader in §5.2 handles both. The tokenizer is the base model's, Falconsai/intent_classification at the pinned revision.


17. Versioning and relation to v2

v1 (this model)v2.0
BackboneDistilBERT, 128 tokensModernBERT-base, trained up to 512 tokens
Specialists1 (the original Falconsai head, reused)3 (intent, code, reasoning), each distilled from its own teacher
GateOne, with three thresholdsOne per specialist, found by coordinate-ascent search
Training dataSupport intents, general intents, multiple-choice QAAdds 11 code tasks and 7 reasoning sources
Routing codeIn the notebook (reproduced in §5.3)Shipped in falconsproof_modeling.py

Changelog

  • v1.0: first release, trained with the small preset (seed 42). Frozen Falconsai specialist with distillation, listwise cross-encoder generalist, temperature scaling, and a single step-function gate found by exhaustive search.

18. Glossary

TermMeaning
StateThe text the decision is about
DecisionOne question with its options
SpecialistThe original 15-intent classifier head
GeneralistThe cross-encoder option head, which handles any options
CoverageThe share of the specialist's belief that falls on the intents behind the offered options
Mapping confidenceHow sure the specialist is about which intent an option means
CollisionTwo options mapped to the same intent
FanoutSeveral questions about one state, answered in the same two forward passes
Held outNever seen in training, calibration or gate tuning
ECEExpected calibration error: the average gap between confidence and accuracy

19. Citation and references

@misc{falconsai_proof_v1_2026,
  title        = {falconsproof v1: a specialist-generalist decision model with a step-function gate},
  author       = {{Falconsai}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Falconsai/proof_V_1}},
  note         = {Fine-tuned from Falconsai/intent_classification}
}

Methods

  • Hinton, Vinyals and Dean (2015). Distilling the Knowledge in a Neural Network.
  • Li and Hoiem (2016). Learning without Forgetting.
  • Sanh et al. (2019). DistilBERT, a distilled version of BERT.
  • Nogueira and Cho (2019). Passage Re-ranking with BERT.
  • Cao et al. (2007). Learning to Rank: From Pairwise Approach to Listwise Approach.
  • Yin, Hay and Roth (2019). Benchmarking Zero-shot Text Classification.
  • Guo et al. (2017). On Calibration of Modern Neural Networks.
  • Geifman and El-Yaniv (2017). Selective Classification for Deep Neural Networks.

Datasets

  • Larson et al. (2019). An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (CLINC150).
  • Casanueva et al. (2020). Efficient Intent Detection with Dual Sentence Encoders (Banking77).
  • Talmor et al. (2019). CommonsenseQA.
  • Mihaylov et al. (2018). Can a Suit of Armor Conduct Electricity? (OpenBookQA).
  • Clark et al. (2018). Think you have Solved Question Answering? Try ARC.
  • Bitext. Customer Support LLM Chatbot Training Dataset.

20. Model card authors and contact

Written by the Falconsai team. Please report issues, misroutes or evaluation results through the Community tab of this repository.


This card is generated from the surgical record itself; the package's lineage.intoto.jsonl is the signed source of truth (verify it free at the Surgeon's public verifier or with the bundled verify_attestation.py).

Architecture

  • Identification: NLP · Small Language Model (SLM) (95% confidence)
  • Source format: safetensors · Intended task: not declared
  • config.json: the repo's config.json, edited
  • Source license: apache-2.0
  • Lineage chain: 2 surgeries — prior signed by ed25519:70d5116dbd76 (FALCONS.AI Model Surgeon V7.96; 4 earlier operation(s) carried) · Falconsai/proof_V_1
  • Post-surgery totals: 67,556,476 parameters · 109 tensors
  • Compute estimate: 5.638717 GFLOPs (comparison metric, not a measurement)

Provenance & operations

  • Parents: Falconsai/proof_V_1/model_edited.safetensors
  • Operations performed: load×1
  • Weight merges recorded: 0
  • Quantized tensors (F32→F16): 0

Surgery Log (ordered)

  1. load — hub:Falconsai/proof_V_1/model_edited.safetensors (270.2 MB, safetensors)

Validation

  • Tissue imaging: not run
  • Structural integrity is testable offline via the packaged load_and_test.py.

Compliance note

The signed attestation + this card together document model composition, modification history, and validation evidence — the record structure technical-documentation obligations (e.g. EU AI Act Annex IV) ask for. This is evidence, not legal advice.


Operated with Model Surgeon — verify this package at https://surgeon.falcons.ai/verify © 2026 FALCONS.AI — Model Surgeon record format. The model weights remain their owner's.

Как запустить

На своей GPU в ailist. Цена — по текущим предложениям провайдеров

GPU в ailist

Универсальный рантайм · transformers · text-classification

Нельзя запустить

Архитектура falconsproof неизвестна transformers в образе рантайма.

Класс архитектуры
FalconsProofModel
Ревизия весов
7112dfb

0 моделей