Victoria

44% of the experts cut (512 to 288 per layer), retrained at 4-bit, and still 70.0% on Terminal-Bench 2.1 (avg@3).

Victoria is a compressed, 4-bit derivative of Qwen/Qwen3.8-Flash-Next. 44% of its experts were removed (512 down to 288 per layer) and the model was then retrained at 4-bit to restore quality, so it runs on a single GPU while keeping most of the original model's coding and agent ability.

See also: Maple, a Canadian-first model built on Victoria.

Highlights

• Fast decode. 5.9B active parameters per token, the same as Qwen3.8-Flash-Next. With the included draft head on, the NVFP4 build decodes 279.6 tok/s on one stream on one NVIDIA B300, 2.08x its 134.7 tok/s with the head off.
• 70.0% on Terminal-Bench 2.1, avg@3. Three full runs of all 89 tasks at 8 hours per task scored 75.3%, 68.5% and 66.3%. The previous NVFP4 build scored 62.5% on the identical protocol.
• Size. 48.0 GiB of weights including the draft head, plus the model's 95.4 GiB n-gram lookup table. The GGUF build holds 49.17 GiB of weights on llama.cpp.
• Fewer wasted tokens. The new build used 35% fewer output tokens than the previous NVFP4 build (69.4M against 107.3M across three runs).
• Two runtimes. NVFP4 for vLLM on NVIDIA Blackwell GPUs, and GGUF Q4_K_M for llama.cpp.

Builds

The two builds are different checkpoints: the NVFP4 build is the latest checkpoint, and the GGUF build is the checkpoint from the first release. Each build's scores were measured on that build only.

Quick start

Download:

hf download rmonsurate/Victoria --include "nvfp4/*" --local-dir .

# full GGUF set, 155.20 GB
hf download rmonsurate/Victoria --include "gguf/victoria-s410-bitexact-0*" --local-dir .

# smaller download, 107.20 GB: the model's lookup table is stored at 8-bit, GPU memory use is the same
hf download rmonsurate/Victoria --include "gguf/victoria-s410-bitexact-tbl8-*" --local-dir .

nvfp4/SHA256SUMS lists the sha256 of every file in the NVFP4 build.

NVFP4 on vLLM, one GPU, draft head on. This is the command the decode speeds below were measured with:

vllm serve /path/to/nvfp4 \
  --served-model-name victoria \
  --tensor-parallel-size 1 \
  --max-model-len 20480 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.80 \
  --enable-prefix-caching \
  --compilation-config '{"inductor_compile_config":{"triton.autotune_at_compile_time":false}}' \
  --kernel-config '{"enable_flashinfer_autotune":false}' \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

• The --compilation-config line is required: compile-time Triton autotuning on this model either hangs or blows out startup time.
• For long agent sessions, the Terminal-Bench runs used --max-model-len 262144 --max-num-seqs 64 with everything else the same, minus --speculative-config.
• We ran vLLM from the Qwen3.8-Flash-Next vLLM image with two fixes backported for a prefix-caching crash on this architecture (vllm#53798 and vllm#54076, open upstream when we built it). On a vLLM without those fixes, leave out --enable-prefix-caching.

GGUF build: use our llama.cpp build

The GGUF files include the model's draft head, stored as 32 extra tensors. Mainline llama.cpp does not know about this head yet, so it refuses to load the files with an error like wrong number of tensors; expected 1256, got 1224. Your download is fine. The files are complete.

The easiest fix is a prebuilt llama.cpp from rmonsurate/llama.cpp-victoria. It is stock llama.cpp b11276 plus one patch for the draft head, with zips for Windows (CUDA and CPU), Linux (CUDA and CPU) and macOS. The same files are on the GitHub release. We are working on getting the patch into mainline llama.cpp.

Run it with the draft head on:

llama-server -m gguf/victoria-s410-bitexact-tbl8-00001-of-00003.gguf -ngl 99 -fa on -c 8192 --spec-type draft-mtp --spec-draft-n-max 3

Or with the draft head off:

llama-server -m gguf/victoria-s410-bitexact-tbl8-00001-of-00003.gguf -ngl 99 -fa on -c 8192

On an M3 Max with 128 GB, one request at a time, the head took generation from 26.8 to 27.8 tok/s (off) to 34.3 to 38.0 tok/s (on), with 70.4% of drafted tokens accepted. We have not measured it on CUDA yet. For the scores on this card we ran 8 slots at 16384 context each across two GPUs, with speculative decoding off.

If you would rather build it yourself, clone the branch:

git clone --branch qwen4exp-draft-mtp https://github.com/rmonsurate/llama.cpp
cd llama.cpp

# NVIDIA GPU (CUDA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# or CPU only, or Metal on a Mac
# cmake -B build
# cmake --build build --config Release -j

The server binary lands in build/bin/ (on Windows, build\bin\Release\). You can also apply victoria-draft-mtp-b11276.patch from the repo above to your own checkout with git am.

Evaluation

NVFP4 build

Counting passes out of three runs per task, this release does better than the previous build on 18 tasks and worse on 6, and ties on the rest (two-sided exact sign test, p = 0.023). Harbor 0.22.0 with the Terminus 2 agent, 8 hours per task, 262144 context, all 89 tasks in every denominator; a trial with no verdict counts as a fail.

HumanEval on the NVFP4 weights is 97.0% (159/164): one sample per problem, thinking on, temperature 0.6, top_p 0.95, top_k 20. That is a single run, not an average over five runs like the GGUF figure, so the two numbers are not directly comparable.

Decode speed and the draft head

One NVIDIA B300, one stream, the vLLM command above, 50 fixed prompts (20 coding, 30 agentic), temperature 0.6, top_p 0.95, thinking off, up to 3 draft tokens per step, a fresh server for each run. Acceptance is accepted draft tokens over proposed draft tokens. The draft head only proposes tokens and the main model verifies every one, so it changes speed, not answers.

GGUF build

The last two rows come from Qwen's Qwen3.8-27B model card and were not run on our setup, so treat them as a rough guide.

Setup: Harbor 0.22.0 with the Terminus 2 agent, 8 hours per task, avg@1, all 89 tasks in the denominator, 262144 context. One avg@1 run over 89 tasks carries a standard error of roughly 5 points. HumanEval on the same weights is 93.2% avg@5, sd 1.3, at n=164. We evaluated text only.

Training curves

Drawn from the training logs of the NVFP4 build (steps 510 to 810) and of the draft head.

Training loss.

Held-out divergence from the full model, agentic and coding slices. Lower is closer to the original model.

How much the model leans toward ending its turn while still thinking. Lower means fewer empty agent turns.

Draft head training: divergence from the main model and draft acceptance rate.

License

Victoria is a derivative work of Qwen3.8-Flash-Next by Farpoint Technologies and carries the same license, the Qwen Community License 1.0, whose full text ships in this repo as LICENSE. Two conditions travel with it: the copyright and permission notice must be included in all copies, and a licensee running a Model as a Service or AI Work Assistant business must obtain a separate license from Qwen before any commercial use. Read the license text and talk to Qwen if that describes what you are building. Attribution goes to Qwen for Qwen3.8-Flash-Next and to Farpoint for Victoria.

Acknowledgements

This work was made possible by the support of Dell, who provided lab access to the 8x NVIDIA B300 system it was built and evaluated on, and of Mario at Buzz (NASDAQ: HIVE). Thanks to the Qwen team for Qwen3.8-Flash-Next, and to the maintainers of llama.cpp, vLLM, harbor and Terminal-Bench.