Qwen3.8-27B-Uncensored-Genesis-V1-MTP — NVFP4 GGUF

An NVFP4 quantisation of LuffyTheFox/Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF, built to Luffy's requested per-tensor quant policy and verified running on an RTX 5070 Ti (Blackwell) with native FP4 support enabled.

@LuffyTheFox — this is yours, take it. Mirror it into your repo, or say the word and I'll open a PR against it directly.

It was actually run, not just produced

Full offload on a 16 GB RTX 5070 Ti, llama.cpp build b10666:

system_info: CUDA : ARCHS = 750,800,860,890,900,1200,1210 | BLACKWELL_NATIVE_FP4 = 1
load_tensors: offloaded 66/66 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   682.03 MiB
load_tensors:        CUDA0 model buffer size = 14114.16 MiB
ftype      : NVFP4

At -c 4096, fully offloaded: ~285–340 t/s prompt, ~42–46 t/s generation.

Quality was checked by running the model's output, not reading it. Asked for a thread-safe O(1) LRU cache with an explicit doubly-linked list and no OrderedDict, the generated code passed: correct eviction order (evicting the genuinely-least-recent node, not the just-touched one), the capacity-0 case, ValueError on negative capacity, and 8 threads x 4000 operations with the linked list still exactly in sync with the dict afterwards. A damaged quant does not write a correct sentinel-node LRU with __slots__ and worked-out lock granularity.

Quant policy

Luffy's policy was written for the MoE Qwen 3.6 35B, which has ffn_down_exps. This model is dense, so that tensor does not exist here; the equivalent is ffn_down. Applied that way, every line resolves:

Verified in the finished file — all four F16 tensors are exactly the intended ones, and the three explicit NVFP4 tensors are NVFP4.

One caveat worth stating plainly: blk.0.attn_gate / attn_qkv were q8_0 in the source, so F16 there stores dequantised q8 values. It does the intended job (those tensors keep q8-level fidelity instead of dropping to 4 bits) but it is not true F16 precision, and it costs more bytes than the source did.

How it was made

The interesting part: no code changes and no rebuild were needed. A stock llama-quantize can already do this.

llama-quantize does not expose NVFP4 as a base CLI ftype — QUANT_OPTIONS in tools/quantize/quantize.cpp has no entry for it, and llama-quant.cpp has no ftype→type case. The obvious read is that you must patch and rebuild. You don't, because of three things in the existing code:

1. --tensor-type resolves its type argument through ggml_type_name() over all ggml types, so =nvfp4 already parses.
2. The override list is vector<pair<regex, ggml_type>> and is matched first-match-wins, so a trailing .* catch-all is expressible.
3. Overrides apply whenever the base ftype is quantised (ggml_is_quantized(default_type)), so any quantised base works — the catch-all then replaces it everywhere.

So the base ftype is a placeholder that never survives, and the real policy is the regex list. Specific patterns first, catch-all last:

llama-quantize \
  --allow-requantize \
  --tensor-type "blk\.0\.attn_gate\.weight=f16" \
  --tensor-type "blk\.0\.attn_qkv\.weight=f16" \
  --tensor-type "blk\.0\.ffn_down\.weight=f16" \
  --tensor-type "blk\.13\.ffn_down\.weight=f16" \
  --tensor-type "blk\.0\.ssm_out\.weight=nvfp4" \
  --tensor-type "blk\.1\.attn_gate\.weight=nvfp4" \
  --tensor-type "blk\.1\.attn_qkv\.weight=nvfp4" \
  --tensor-type ".*=nvfp4" \
  Qwen3.8-27B-Uncensored-Genesis-V1-Q8_K_P.gguf \
  Qwen3.8-27B-Uncensored-Genesis-V1-NVFP4.gguf \
  Q8_0

Then the file type label, which is the one thing the above gets wrong. The header inherits the placeholder base ftype and reports file type = Q8_0. That is cosmetic — loading dispatches on per-tensor types — but it is wrong on the shelf, so patch the single uint32 in place (no 15 GB rewrite):

python gguf-py/gguf/scripts/gguf_set_metadata.py model.gguf general.file_type 39 --force

39 is LLAMA_FTYPE_MOSTLY_NVFP4. The loader guesses the ftype correctly from the majority tensor type on its own (GGML_TYPE_NVFP4 → LLAMA_FTYPE_MOSTLY_NVFP4 is already in that switch in llama-model-loader.cpp) and only overrides the guess when general.file_type is present — so a stale value is actively worse than none. After the patch it reads ftype : NVFP4.

Notes on the process

• Requantised from Q8_K_P, not from BF16 — no BF16 source was available. It is lossy-on-lossy for the tensors that were genuinely q8, though a substantial number of source tensors are bf16 and requantise cleanly.
• Luffy's writeup describes dequantising Q8_K_P to FP16 first, then requantising. That intermediate step is unnecessary and slightly harmful: --allow-requantize already dequantises each block to f32 before quantising, so routing through F16 only adds a rounding step.
• --dry-run resolves the whole policy without writing anything, which is how the missing ffn_down_exps was caught before spending any time.
• tensor_type_fallback handles tensors whose shape isn't divisible by the NVFP4 block size, warning per tensor. Here it never fired.
• The quantise itself took 59 seconds on CPU.

Running it

llama-cli -m Qwen3.8-27B-Uncensored-Genesis-V1-NVFP4.gguf \
  -ngl 99 -c 4096 --temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0

Sampling above follows the Genesis guidance (temp 0.6, top_k 20, top_p and min_p disabled). For code, also set --repeat-penalty 1.0: code repeats identifiers by nature, and any penalty pushes the model off the identifier it just chose, which is a reliable way to get generated code that won't run.

Give it room to think

Genesis reasons at length before answering. In testing it spent 900 tokens, then 2600 tokens, entirely inside [Start thinking] without reaching its final answer. That is the model, not the quantisation — but with a small -n it looks like the model is broken or rambling. Budget generously (-n 4096 or more) for anything non-trivial, or you will only ever see it thinking.

Needs a Blackwell card and a llama.cpp built with FP4 support for the fast path (BLACKWELL_NATIVE_FP4 = 1 in system_info); b10666 or newer.

Credits

Base model and quant policy: LuffyTheFox. Genesis V1 finetune: HauhauCS. This repo is only the NVFP4 conversion and the verification that it runs.