Qwen3.8-Flash-Next NVFP4 with NVFP4 PLE
This repository is a serving derivative of
Inferact/Qwen3.8-Flash-Next-NVFP4.
It keeps the ModelOpt NVFP4 backbone unchanged and converts only the BF16
position-learning enhancement (PLE) n-gram table to NVFP4.
This repository contains model weights and Hugging Face metadata only. It does not include runtime code, shell scripts, Docker files, or result artifacts.
The packed PLE format requires an external qwen38-nvfp4-ple runtime plugin.
Stock vLLM and Transformers cannot load it. The checkpoint retains the original
vision tower.
Storage
| Artifact | Exact bytes | GiB |
|---|---|---|
| Source checkpoint payload | 182,738,792,440 | 170.189 |
| Source BF16 PLE table | 102,400,491,520 | 95.368 |
| Converted NVFP4 PLE table | 28,800,138,752 | 26.822 |
| Complete converted checkpoint | 109,138,439,672 | 101.643 |
The conversion saves 73,600,352,768 bytes, or 68.546 GiB. It does not use REAP or remove experts.
Why this fits on one DGX Spark
This release targets one NVIDIA DGX Spark because its GB10 CPU and GPU share 128 GB of LPDDR5x unified system memory. NVIDIA documents the hardware and memory layout. The GPU executes the NVFP4 backbone. The plugin keeps the 26.822 GiB packed PLE table CPU-offloaded and decodes only requested rows to BF16. Both allocations draw from the same physical memory pool.
In a measured full multimodal TP1 run, vLLM reported 73.97 GiB during model
loading and reserved a 2 GiB KV cache. The vision tower added 0.85 GiB over
text-only serving. The PLE worker has been tested only with vLLM's mp
executor at TP1.
API startup took about 639 seconds. After image and text requests, the host had 19.8 GiB available with 14.0 GiB of system-wide swap in use. The default 4,096-token context, single sequence, and 2 GiB KV cache are conservative. This demonstrates fit; it is not a throughput claim.
External runtime contract
| Component | Required or tested value |
|---|---|
| vLLM image | vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8 |
| vLLM version | 0.1.dev20073+g8e685d198 |
| PLE mode | CPU offload with BF16 lookup output |
| Serving model | Full Qwen4ExpForConditionalGeneration wrapper |
| Multimodal limits | One image, no video, 65,536-to-262,144-pixel budget |
| Executor | mp, including at TP1 |
| Validated GPU | NVIDIA GB10, compute capability 12.1 |
The pinned vLLM uni executor does not spawn the PLE CPU worker. The external
plugin must select mp and register itself inside vLLM's special PLE
subprocess.
PLE format
Format qwen38-nvfp4-ple-v1 stores each group of 16 values as packed E2M1
values, one E4M3 block scale, and one FP32 global decode scale per source PLE
shard. The runtime gathers only requested rows and decodes those rows to BF16.
It never expands the complete table.
Evaluation
A controlled teacher-forced study used 32 calibration prompts and eight held- out prompts, covering 236 next-token positions. The executable reference was the official FP8 backbone with normal PLE behavior.
These measurements isolate changes to the PLE representation. They do not measure the separate quality difference between the Inferact NVFP4 backbone and the original model. This uploaded checkpoint uses the first row below; the REAP rows are comparison experiments only.
| Perturbation | Top-1 agreement | Reference top-1 in candidate top-5 | KL, nats | NLL delta, nats | Logit cosine |
|---|---|---|---|---|---|
| NVFP4 PLE only | 91.525% | 100.000% | 0.015591 | +0.019442 | 0.995399 |
| REAP only, 25% experts removed | 87.712% | 100.000% | 0.042569 | +0.043668 | 0.991027 |
| REAP and NVFP4 PLE | 91.102% | 99.576% | 0.040298 | +0.040545 | 0.990824 |
The converted checkpoint passed text-only and full multimodal vLLM smoke tests. The multimodal run loaded the vision tower and served PNG inputs. With a neutral prompt, it correctly identified a generated Moon scene, Earth, casual clothing, and the absence of a spacesuit.
Limits
- The alignment study is not a task-quality benchmark.
- The runtime checks are smoke tests, not throughput or vision benchmarks.
- Long-context, concurrent-request, tensor-parallel, and expert-parallel paths have not been validated.
- The validated image path accepts one image within the default pixel budget. Video is disabled by default and has not been validated.
- The source MTP weights remain in the checkpoint, but the validated runtime did not enable MTP.
- The tested host had system-wide swap in use after inference.
- The PLE decoder uses ordinary PyTorch CPU operations. A fused decoder could reduce lookup latency.
Provenance and license
The backbone comes from
Inferact/Qwen3.8-Flash-Next-NVFP4,
which derives from
Qwen/Qwen3.8-Flash-Next.
Only the PLE representation changed here.
The model weights remain under the Qwen Community License 1.0 in LICENSE.