Prismyra decision FP8, 36 layers, for Qwen3.6-35B-A3B-FP8

A full-weight FP8 checkpoint truncated to 36 of the base model's 40 layers, with a LoRA folded into the FP8 weights and requantized per 128x128 block. It is for Prismyra: an engine that answers typed questions (booleans, choices) about a document by reading the probability of each declared option's token in a single forward pass, rather than generating an answer -- see the Prismyra repository for what that trades off and where it does not apply.

The adapter folded into this checkpoint was originally distilled from the full 40-layer checkpoint's own output distribution (see "How it relates to the other checkpoints" below), and every revision since has refined it further by self-distillation: training against the immediately-preceding best 36-layer checkpoint's own forward pass. For this revision, that means the previous revision's own output, not this revision's -- a revision cannot be its own teacher before it exists.

What changed in this revision, in one line: a held-out panel of public benchmarks this checkpoint had never been evaluated on turned up two judgment types it loses to a comparable general-purpose model on -- sentiment classification and causal/multi-step reasoning -- and this revision mixes four licence-clear public datasets covering both types into the training pool at a small, fixed fraction (4.00% of the combined pool, 2,241 of 56,026 rows), closing the sentiment gap with no measured regression on this project's production-safety set or any other evaluation set at the confidence levels used. The adapter's shape, target modules, initialization and hyperparameters are unchanged from the previous revision.

The fraction mattered: an earlier attempt at mixing the same four families in at roughly 7.4% improved sentiment by more but also regressed the production-safety set below; halving it to 4.00% removed the regression and kept the sentiment gain (see Training and Evaluation).

The previous revision (the 1.71x-row-growth revision described in earlier editions of this card) stays available at commit 88d8f2b236c20ee897289c3a79705d2432883690 (pip install "prismyra[server,fast] @ git+https://github.com/littlemex/Prismyra@v0.4.6" is the matching engine version). Every number below for "the previous revision" is that commit's own previously-published measurement, reused as-is rather than re-run, except where this card's own Evaluation section below reports a fresh, revision-to-revision comparison.

What this means in practice: on a held-out panel of two public sentiment benchmarks (400 questions total, neither used in training), this revision answers significantly more questions correctly than the previous revision (sign test, Holm-corrected, p = 0.0017). On a held-out panel of two public causal/multi-step-reasoning benchmarks (400 questions, also not used in training) the difference points the same direction but does not reach significance at this sample size. On the project's production-safety set (764 questions) and twelve further held-out and development sets, no item was found to have clearly regressed (every item's 95% confidence interval against the previous revision either favoured this revision or straddled zero). Confirmed on two independently built 10,000-question sets, each opened and scored exactly once, on the production NVFP4 serving path: both are within a ±0.5-point non-inferiority bound of the previous revision (see Evaluation).

Use

pip install "prismyra[server,fast] @ git+https://github.com/littlemex/Prismyra@v0.4.7"

from prismyra import Prismyra, Boolean

engine = Prismyra("littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-36l")
result = engine.ask(document, [Boolean(id="q", prompt="Is this a two-sided agreement?")])

prismyra-serve --model littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-36l --require-kernels

--require-kernels refuses to start rather than serve at a fraction of the speed. Prismyra's fused kernels are matched against the checkpoint by per-layer tensor shape, not by layer count, adapter rank, or how many rows the adapter trained on, so this checkpoint gets every kernel the base model gets regardless of which revision's weights are loaded -- there is no flag to set for this. The Latency section below shows this revision running at the same speed as the previous one, consistent with (though not proof of) both using the same kernels; --require-kernels itself is what actually confirms kernel use, by refusing to start otherwise.

Serving at v0.4.7: /v1/decide, --batcher, and a determinism self-check

prismyra-serve --model littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-36l --require-kernels --batcher

POST /v1/decide answers through autotrust/JEV-27B's own request shape (kind/state/question/options) instead of Prismyra's own /ask shape, for a caller already built against JEV's API. See docs/DECIDE.md for the full contract.

curl -s localhost:8000/v1/decide -H 'content-type: application/json' -d '{
  "kind": "noul",
  "state": "Returns are accepted within thirty days of delivery. Unopened items are refunded in full.",
  "question": "Is there a thirty day limit?"
}'

{"id":"0","kind":"noul","options":["false","true"],"probabilities":[0.001131512108258903,0.998868465423584],
 "choice_index":1,"choice":"true","value":true,"adaptation":"native","symbols":null,
 "protocol":"prismyra-decide-v1","model":"littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-36l","merged_with":[]}

This is request-shape compatibility for the subset of JEV's protocol this checkpoint can serve (choice up to 255 options; JEV's own ceiling is 256) -- it is not a claim that the two servers' probabilities agree; see docs/DECIDE.md for what each one computes.

--batcher holds several callers' documents on the device at once (prismyra.schedule.Batcher), evicting least-recently-used documents to make room for a new one. As of v0.4.4 that eviction accounts for page-level fragmentation, not only a token-count sum, so a document the plain unbatched queue would answer is never refused merely because the shelf's own eviction history happened to fragment its page pool -- verified bit-for-bit against /ask's own answer on real hardware. See prismyra/paged.py and prismyra/schedule.py.

A construction-time FP8 determinism self-check calls the installed FP8 kernel on a fixed input eight times in the process about to serve requests, and refuses to start if any call disagrees with the first. This is a bounded probe, not a proof of determinism: eight agreeing calls make a disagreement observable when it is frequent enough to show up in eight tries, and nothing stronger. On by default (PRISMYRA_WITHOUT=fp8_determinism turns it off); confirmed passing on this checkpoint's own card (L40S), recorded in engine.applied.notes. See prismyra/kernels/qwen3_moe.py. This check is separate from the NVFP4 routed-expert tactic-pinning mechanism described in docs/KERNELS.md, which this checkpoint's own weights do not use.

Training

The adapter targets the same 250 modules as the previous revision (attention and gated-delta-net projections, the shared expert) at rank 64, alpha 128, and starts from the same padded-up rank-16 weights the previous revision also started from (not from the previous revision's own trained weights). Capacity, target modules and initialization are all held fixed relative to the previous revision.

What changed: training data is the previous revision's own 31,418-row pool plus 22,367 additional rows, for 53,785 total (1.71x). The added rows come from 9 of the 13 source families the previous revision drew on -- OpenBookQA, SciQ, CommonsenseQA, AQuA-RAT, MMLU auxiliary-train, Cosmos QA, MNLI, QQP and Dream -- sampled from a later window of the same per-family shuffle the original pool used, so the two windows do not overlap. The other 4 families (ARC, a reading-comprehension set called QuALITY, a BIG-bench logical-deduction task, and a HotpotQA comparison slice) were left untouched because earlier work had already drawn each of them down close to its practical limit. Every added row was checked against every evaluation set and every other training artifact in this project for overlap, using the same three-stage check the project uses throughout: exact full-text match, then 13-word n-gram overlap, then whole-string containment for short fields. One side effect of adding rows from 9 of 13 families: optimizer steps rise by about the same 1.71x (one epoch, same per-step batch), and the family mix shifts toward those 9 relative to the previous revision's pool.

The distillation target for every row, old and new alike, is the previous revision's own 36-layer forward pass (the self-distillation step described above). The previous revision's 31,418 original rows only carry this signal on 68.0% of them (21,362 rows) -- an artifact of that pool having been assembled incrementally across several earlier training passes, not a deliberate choice -- while all 22,367 added rows carry it on every row, so the signal's overall coverage rises to about 81.3% of this revision's full training set. As with the previous revision, roughly 7% of rows (both carried-over and added) take their gold label from a frontier LLM (Kimi K3) instead of the pool's own label, used only where Kimi's single-letter answer disagreed with the pool's; no rationale text from Kimi is reproduced in this checkpoint's weights or in this card.

One epoch, learning rate 2.5e-5, four L40S GPUs. The previous revision's own training (31,418 rows, three L40S GPUs) took about 18.9 GPU-hours (378.0 minutes of wall-clock time, from that run's own training log, times three GPUs). This revision's own training log reports 431.8 minutes of wall-clock time on four L40S GPUs, for about 28.8 GPU-hours -- more than the previous revision's 18.9, but less than a simple proportional estimate for 1.71x the rows (about 32 GPU-hours) would suggest, consistent with this revision's per-GPU throughput (0.52 rows/second, against the previous revision's 0.46) being a little higher.

On a fixed 2,500-row slice of the training distribution, held out of the gradient updates and used identically to measure the previous revision, this reduced the KL divergence to the distillation target by 52.5%, against 51.2% for the previous revision -- both past this project's bar of cutting that gap by at least half relative to the untrained 36-layer checkpoint, with this revision cutting a little further (no interval was computed for this 1.3-point difference; read it as "similar," not as a separately significant result).

This revision adds one more training ingredient on top of the previous revision's 53,785-row pool described above. A held-out panel of 18 public benchmarks (3,233 questions, checked for overlap against this project's full training, calibration and evaluation corpus and found clean) was used to find judgment types this checkpoint family loses on against a comparable general-purpose model. Two came back significant: sentiment classification and causal/multi-step reasoning; the other nine judgment types in that panel were a tie or a win, and every one of them already had a matching family in the training pool -- neither losing type did. build_weak_category_mixture.py (in the project's recipes/decision-lora directory) adds four licence-clear public datasets covering the two missing types -- zeroshot/twitter-financial-news-sentiment (MIT) and cardiffnlp/tweet_eval's sentiment config (CC BY 3.0) for sentiment; pkavumba/balanced-copa and allenai/social_i_qa (both CC BY 4.0) for causal and social-situation reasoning -- and mixes 2,241 rows from them into the existing pool for 56,026 rows total (4.00% of the combined pool). An earlier attempt mixed the same four families in at roughly 7.4% (4,288 rows): it improved sentiment by more but also measurably regressed the production-safety set below, concentrated on items either topically close to the new rows or already a close call before training -- not a data leak (a three-stage overlap check found the two sets share no row), but ordinary interference from the one thing the new rows lacked that every pre-existing row has: a self-distillation target holding the step toward this checkpoint's own prior output as well as toward the gold label. Halving the fraction to 4.00%, with nothing else changed, removed the regression and kept the sentiment gain (see Evaluation); the new rows still train on gold labels only, with no self-distillation target of their own, same as the larger-fraction attempt. One epoch, same per-step batch and learning rate as above, four L40S GPUs, 449.0 minutes of wall-clock time, for about 29.9 GPU-hours.

Evaluation

Point differences are accuracy percentage points; intervals are a paired bootstrap (4,000 resamples, paired by question) at 95%; p-values are two-sided sign tests on the per-question pairs where the two revisions disagree, Holm corrected where more than one judgment type is tested at once. Every number below is this revision measured fresh against the previous revision (commit 88d8f2b2), served by Prismyra on one L40S.

The two judgment types this revision's new training rows target, 400 held-out questions each, neither used in training (two public sentiment benchmarks for the first row; two public causal/multi-step-reasoning benchmarks for the second), this revision vs. the previous revision:

The sentiment gain is the one this revision's new rows were added to close; the reasoning gain points the same direction but did not reach significance at this sample size.

Fourteen further held-out and production item sets, checked for "did any item clearly regress" rather than for a gain -- a development set (1,893 questions), four production reading sets (RACE 579, BoolQ 400, and the same RACE questions buried in roughly 7,000 and 10,000 tokens of unrelated text, 157 questions each), the project's production-safety transfer set (Kev, 764 questions, roughly 30% from families this checkpoint has never trained on in any form), and eight further held-out judgment types this checkpoint ties the comparable general-purpose model on (natural-language inference, commonsense, world knowledge, arithmetic, content review, tool use, reading comprehension, code):

"Regressed" here means the item's 95% confidence interval against the previous revision falls entirely below zero; none of the fourteen do, including the production-safety set, even though its own point estimate (-1.047) is the largest negative one in the table -- its interval still crosses zero. An earlier attempt at a larger mixed-in fraction (7.4%) did regress this exact production-safety set by this exact test (its interval fell entirely below zero); that attempt was not shipped (see Training).

Final confirmation, on two independently built, 10,000-question sets (72% production-style questions, 8% long-document questions, 20% contrast/adversarial questions each), each opened and scored exactly once, on the production NVFP4 serving path (see the NVFP4 sibling checkpoint's own card for that engine):

Both sets land essentially tied with the previous revision, well inside the ±0.5-point non-inferiority bound used to confirm this revision before it shipped; two questions in set B were excluded on both revisions for the same reason (a branch exceeding this checkpoint's token budget).

This checkpoint is also measured against JevBench (a 231-question benchmark none of these checkpoints trained on) and against four general-purpose LLMs on five long-standing sets, in the training recipe's README. The short version, measured on an earlier revision of this checkpoint and not re-measured for this one: this checkpoint's family is not significantly different from the 40-layer checkpoint on JevBench, but its calibration there (Brier score and expected calibration error) is worse than the 40-layer checkpoint's; and against general-purpose LLMs it loses on accuracy overall, most clearly on Kev, at one to a few orders of magnitude lower estimated GPU-time cost depending on how long the context is. See "Compared with other decision models" below for two more decision models measured the same way, and the frontier-model numbers behind the paragraph above -- none of those comparisons have been re-run for this revision, since this revision is not shown to differ from the previous one outside the sentiment judgment type described above.

Latency

One ~5,300-token document, repeated across calls, 1/16/64 questions asked about it in a single call. Same L40S, v0.4.0's default construction (Prismyra(model) -- batch-invariance is claimed unconditionally for every CUDA engine since this release, and the layer-interleaved fusion this project calls interleaved_fork is on by default), 15 rounds alternating the three widths, the first 4 discarded as warm-up:

Medians of 11 rounds each (0.5-1.4% spread). The two revisions still agree with each other to within 0.6% at every width -- training on more rows still does not move these numbers, which is the same conclusion the previous measurement reached. The absolute numbers above are higher than this card previously reported (96.8/204.2/411.4 ms): that earlier measurement did not build its engine with the construction this project's own default now uses, and did not include the layer-interleaved fusion this release adds by default. The earlier numbers are not comparable to this release's default and are not restated here.

Separately, and not re-measured for this revision: the previous revision was also measured against the 40- and 32-layer checkpoints and the untrained base, same script and GPU, on the same ~5,300-token document:

The speed difference across the three published depths tracks their layer count, which is what fused kernels applying at every depth looks like -- a checkpoint that fell back to the unfused path would be roughly three times slower than the figures above, at any depth. This table was measured on a construction that predates both v0.4.0's unconditional batch-invariance and its interleaved_fork fusion, so it is not comparable to the figures in the table above and no comparison between the two is drawn here.

Compared with other decision models

Measured against two more decision models the same way as the comparisons above: same gold labels, same 2,057 questions across the five sets this project's own README describes, each model read through its own native API rather than forced through the other's. JEV-27B-VL's FP8 quantisation (Atlas3D/JEV-27B-VL-FP8, a community release) stands in for the unquantised 27B original, which does not fit a 48 GB-class card. Reproduce the speed half with benchmarks/compare_jev_d1.py; the two competitors' own numbers below were read through their own serving code, not through this script.

Accuracy (sign test per set, Holm-corrected across the five; "tie" means not significant after correction):

Neither competitor beats this checkpoint on accuracy on any of the five sets measured; the only significant win over JEV-27B-VL specifically is on Kev, the hardest of the five transfer sets.

Probability quality (2,057 questions, lower is better for both):

This checkpoint wins on Brier against both; both competitors are better calibrated by ECE, d1-3B markedly so -- reported rather than smoothed over, since this checkpoint is confident more often than it is right by this one measure.

Speed (one document held at 3,401 tokens, 1/16/64 questions in a single call, median of 7, 2026-10-10). The one-question row is a same-day, same-GPU (L40S) measurement interleaved with JEV-27B-VL's own run; the 16- and 64-question rows for this checkpoint are from the same session's own version-to-version comparison on the identical GPU. JEV-27B-VL's 16- and 64-question figures and all of d1-3B's figures are not from that session -- they are carried over from an earlier comparison round on the same hardware class, reused because neither competitor's served checkpoint has changed since that round:

JEV-27B-VL is 9.2% faster at one question -- a same-day, same-GPU result (170.5 ms against 154.8 ms). The cause is not established: a candidate explanation (a tuned kernel setting silently not applying) was checked directly on hardware and ruled out. At 16 and 64 questions this checkpoint still wins by a wide margin, because JEV's own API answers one decision per call and pays for the document again on every one -- a design limit of that API. d1-3B is faster at every width measured: a 3B dense model with no routed experts has less to compute than a 35B-A3B MoE at any width, which is the honest trade this checkpoint's accuracy numbers above are made against.

Determinism: JEV-27B-VL is not bit-deterministic -- the same input, read repeatedly in one process, across companions in one batch, and across process restarts, returned at least three distinct probability values in an eleven-trial sample. d1-3B was bit-exact across five process restarts and every batch-companion pattern tried. This checkpoint's own construction-time self-check (see "Serving at v0.4.7" above) targets one hardware-level gap in its own determinism story that an unseeded kernel probe could not see; it is a bounded probe confirmed on this checkpoint's own card, not the broader empirical guarantee d1-3B's measurement above demonstrates.

Memory: this checkpoint holds about 32.7 GiB on an L40S; JEV-27B-VL's FP8 quantisation holds about 34.2 GiB (similar, despite the dense-27B-against-MoE-35B difference, because JEV also carries a vision encoder these text-only questions never use); d1-3B holds about 6 GiB on either card, light enough to run several copies beside everything else on one GPU.

Against frontier closed models, measured the same way on the same five sets in an earlier comparison round (pooled accuracy for this checkpoint in that round, 90.71%, differs slightly from the 91.40% above because it predates this card's current adapter revision): Claude Opus 5.5, Claude Fable 5.1 and GPT-6 Astra all score significantly higher overall (95.28%, 93.53%, 93.68% respectively), most clearly on the Kev transfer set; only Claude Haiku 4.5 (83.71%) scores lower. What is traded for that accuracy is cost -- the closed models' per-1,000- question Bedrock price ranges from about 4x to about 2,000x this checkpoint's own GPU-time estimate, the gap widest on the long-context sets. See recipes/decision-lora/README.md's own table for the full comparison.

How it relates to the other checkpoints

This project publishes five repositories:

• littlemex/prismyra-decision-lora-qwen3.6-35b-a3b -- the LoRA adapter for the full 40-layer base model, trained separately from this checkpoint's own LoRA. This is the original teacher the first 36-layer-depth adapter was distilled from, before the self-distillation rounds described under Training took over.
• littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-40l -- all 40 layers, that same adapter folded in.
• littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-36l -- this repository.
• littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-32l -- 32 layers, rank-16 adapter. Faster again, and ahead of both longer checkpoints on the two long-context sets above (bury7k/bury10k, measured on the previous revision), but the one of the three that misses the BoolQ bar used for that repository's own release decision (by a single question) and is not significantly different from the untrained base on JevBench.
• littlemex/prismyra-decision-qwen3.6-35b-a3b-nvfp4-36l -- this checkpoint's routed experts in NVFP4 instead of FP8, for Blackwell GPUs. Routed experts are never touched by this checkpoint's LoRA, so they are identical across every revision of this repository, and that sibling repo reuses them unchanged. This revision's own attention/gated-delta-net/shared-expert weights were separately found to improve that sibling's configuration the same way they improve this one; whether and when that sibling repository is updated to match is tracked on its own and may lag this one.

Rolling back to the previous revision

If this revision's sentiment gain is not relevant to a caller's use case and the caller would rather pin the exact weights measured in earlier editions of this card, the previous revision stays available unchanged at commit 88d8f2b236c20ee897289c3a79705d2432883690:

from huggingface_hub import snapshot_download
path = snapshot_download(
    "littlemex/prismyra-decision-qwen3.6-35b-a3b-fp8-36l",
    revision="88d8f2b236c20ee897289c3a79705d2432883690",
)

Nothing about serving changes between revisions -- same shape, same depth, same kernels -- so no other configuration needs to change to roll back.

Limitations

This checkpoint answers typed questions (booleans, fixed choice lists) about a supplied document; it does not generate free-form text, and the "Use" section's single-pass design is specifically not suited to open-ended questions or to tasks where the right answer is not one of a short, declared list. This revision's sentiment gain over the previous revision is specific to the two public sentiment benchmarks it was measured on and to this checkpoint's typed-question format; it is not shown to hold on free-form sentiment analysis, on sentiment-adjacent judgment types outside those two benchmarks, or outside this project's own evaluation harness. The matching reasoning gain did not reach significance at the sample size tried, so it should be read as a direction, not a confirmed improvement. This checkpoint's calibration (how well its stated probabilities track its actual accuracy) has not been re-measured for this revision; the previous revision's own calibration numbers on JevBench, cited under Evaluation, are worse than the 40-layer checkpoint's.

Hardware

Full-weight FP8 at this depth needs about 32 GB of VRAM for the dense weights and routed experts, which fits on one 48 GB L40S but not on a 32 GB card (see the NVFP4 sibling above for a Blackwell-class 32 GB alternative). All numbers in this card were measured on L40S (sm_89); FP8 block-scaled kernels require a GPU with FP8 tensor-core support.

Citation

See the Prismyra repository for how to cite this project.

Licence and data terms

Apache-2.0, the base model's licence, and this repository carries the base model's LICENSE file unchanged. Some of the training sources behind this checkpoint's LoRA carry their own terms -- RACE, SciQ, DREAM and AG News are distributed for non-commercial research use -- so check those terms before commercial use of this checkpoint. (Earlier editions of this card named only RACE and SciQ; DREAM and AG News carry the same restriction and were missed until the full source list was re-checked against each source's own licence file.) The four datasets added in this revision are licence-clear for commercial use: zeroshot/twitter-financial-news-sentiment (MIT), cardiffnlp/tweet_eval's sentiment config (CC BY 3.0), and pkavumba/balanced-copa and allenai/social_i_qa (both CC BY 4.0, the second not independently re-confirmed against a primary source -- see the link below). About 7% of the previous revision's training rows (both the rows carried over from earlier revisions and the ones added then) take their gold label from a frontier LLM (Kimi K3, called through its provider's API) rather than from the pool's own label, used only as a single-letter answer where it disagreed with the pool; no generated rationale or free text from that model is reproduced in this checkpoint's weights or in this card. The four new datasets added in this revision use their own gold labels throughout, with no frontier-model relabelling. No output of a closed, proprietary model was used anywhere in this pipeline.

Training data and licensing: every family in the training pool, its source, its licence, and the teacher models used for gold-label correction and self-distillation, are listed with the licence text each was read from, in docs/DATA_PROVENANCE.md.