Qwen3.8-27B "MXFP8-CRACK" — NInfer artifact

A .ninfer single-file artifact that lets the NInfer engine (single-GPU C++/CUDA inference, RTX 5090) run the community fine-tune dealignai/Qwen3.8-27B-MXFP8-CRACK.

Download & run

hf download WaveCut/Qwen3.8-27B-MXFP8-CRACK-NInfer crack.ninfer --local-dir models

# CLI, one request:
./build/apps/ninfer models/crack.ninfer --prompt "Summarize the difference between prefill and decode." --max-context 16384

# OpenAI/Anthropic-compatible server:
./build/apps/ninfer-serve models/crack.ninfer --max-context 16384 --kv-capacity auto

Hardware. Two engines run this artifact:

• Upstream NInfer — primary target one RTX 5090 (32 GB); in practice any single Blackwell sm_120 card with 32 GB works (the verification smoke above ran on an RTX PRO 4500).
• Don-Chad/ninfer-3090 (v0.6.0+) — RTX 3090 (24 GB) edition where Qwen3.8-27B is a first-class, tested target: the native SM86 runtime loads the official groupwise .ninfer artifact directly (same qwen3.8-27b / groupwise-int identity as this file), ships run-qwen38-* launch scripts, image understanding, and sustains up to a 171K-token INT8 context on one 3090 (226K with the optional RotorQuant rk8v4 KV cache).

How it was built

The source checkpoint stores weights in two packed 8-bit schemes, so it was first dequantized to BF16 and re-named into the official Qwen3.8-27B tensor layout, then converted with the unmodified stock converter:

1. Text + Vision (582 tensors) — MXFP8: 4×FP8-E4M3 packed per U32 word, one U8 UE8M0 scale (2^(b-127)) per 32-element group → BF16.
2. MTP draft layer (8 tensors) — MLX affine int8: uint8 codes with per-128-group F16 scale and bias, w = q*scale + bias → BF16. The formula was verified against the official Qwen/Qwen3.8-27B MTP weights (cosine similarity ≈ 1.0, σ matches to 5 decimals).
3. Renames: language_model.model.* → model.language_model.*, language_model.lm_head.* → lm_head.*, vision_tower.* → model.visual.*; the vision patch embedding was permuted from the MLX conv layout [O,T,H,W,I] to torch [O,I,T,H,W]. config.json of the fork passes the converter's registered validation unchanged.
4. Frontend resources (tokenizer, chat template, preprocessor configs, generation_config.json) are the official Qwen/Qwen3.8-27B files — the NInfer converter pins them by SHA-256. The fork's tokenizer/chat-template files are byte-identical; its generation_config.json differs only by a no-op key, so official sampling defaults apply unless overridden per request.
5. Standard converter: python3 -m tools.convert.qwen3_8_27b.convert --model <bf16-dir> --out crack.ninfer (82 s on one RTX PRO 4500).

crack.ninfer.conversion.json (included) is the converter's own report for this exact file.

Caveats

• Double quantization. The source is already 8-bit (MXFP8 / affine int8); this artifact re-quantizes into NInfer's groupwise-int profile (Q4/Q5/Q6/W8 groupwise + BF16). Expect marginally lower fidelity than an artifact built from a native BF16 checkpoint of the same tune.
• The engine binds this file to the registered qwen3.8-27b / groupwise-int profile: architecture, tokenizer and chat template are identical to the official model; only the weights carry the fine-tune.
• Upstream NInfer is specialized for one RTX 5090-class device and a single CUDA device; for RTX 3090 use the ninfer-3090 fork (see Hardware above).
• Per the upstream card: set your sampler explicitly (temperature=1.0, top_p=0.95, top_k=20) if you rely on the tune's intended behavior.

Lineage

Qwen/Qwen3.8-27B
  └─ dealignai/Qwen3.8-27B-MXFP8-CRACK   (fine-tune + MXFP8/MLX-affine quant, Apache-2.0)
       └─ BF16 dequant + official-layout rename (adapter described above)
            └─ crack.ninfer  (this repo, NInfer groupwise-int profile)

Licensed Apache-2.0, same as both upstreams.