Related models: all models
Swift-1.5-Qwen3.8-27B-Uncensored-BF16 · Swift-1.5-Qwen3.8-27B-Uncensored-FP8 · Swift-1.5-Qwen3.8-27B-Uncensored-FP8-NInfer · Swift-1.5-Qwen3.8-Flash-Next-NVFP4-FP8PLE · Swift-1.5-Qwen3.8-Flash-Next-Rank2-Abliteration-Patch · Swift-Qwen3.8-27B-FP8 · Swift-Qwen3.8-27B-Uncensored-BF16 · Swift-Qwen3.8-27B-Uncensored-FP8 · Swift-Qwen3.8-27B-Uncensored-NVFP4-LocalHessian-ActivationHeadroom-NInfer
Swift-Qwen3.8-27B-FP8
FP8 quantization of
ukisai/Swift-Qwen3.8-27b.
This is an independent community quantization and is not an official UkisAI release.
The goal of this checkpoint is to preserve the behavior of Swift-Qwen3.8-27B while reducing VRAM requirements and enabling high-throughput inference with SGLang, including speculative decoding with the model's native MTP head or an external DFlash2 draft model.
Model summary
- Upstream model:
ukisai/Swift-Qwen3.8-27b - Architecture:
Qwen3_5ForConditionalGeneration - Quantization format:
compressed-tensors - Quantization scheme:
FP8_BLOCK - Weight block size:
128 x 128 - Activations: dynamic FP8
- Activation group size:
128 - Quantizer:
llmcompressor 0.13.0 - Declared context length:
262,144 - Checkpoint size: approximately
29 GB - Native MTP components: retained
The checkpoint was produced from the BF16 Swift-Qwen3.8-27B model rather than requantizing an already quantized derivative.
Quantization details
The quantization process used the official Qwen3.8 FP8 configuration as a reference for the block-FP8 layout.
Checkpoint audit:
| Item | Count |
|---|---|
| Total checkpoint tensors | 1,199 |
| 2D weight tensors | 617 |
| FP8 quantization candidates | 407 |
| Effectively excluded / preserved modules | 626 |
| Incompatible FP8 candidates after validation | 0 |
Matrices that are not compatible with the required 128 x 128 block
structure were preserved instead of being forcibly quantized.
The following classes of tensors were intentionally preserved where appropriate:
- embeddings
lm_head- normalization parameters
- non-2D weights
- matrices whose dimensions are incompatible with the FP8 block layout
The native MTP layers are retained. Compatible MTP projection matrices are quantized to FP8, while incompatible components remain unquantized.
Validation
Validated locally on:
- NVIDIA RTX PRO 6000 Blackwell 96 GB
- SGLang
0.5.19.dev135+ga4ffb996d compressed-tensors 0.18.0- CUDA-capable Linux deployment
- single-GPU tensor parallelism (
TP=1)
SGLang successfully loads the checkpoint as:
type=Qwen3_5ForConditionalGeneration
quant=compressed-tensors
Observed target-model weight memory during loading:
28.47 GB
OpenAI-compatible /v1/chat/completions inference was validated successfully.
Multimodal inference has not yet been separately benchmarked for this quantized checkpoint.
Performance
All measurements below are local measurements from a single NVIDIA RTX PRO 6000 Blackwell 96 GB GPU.
They are intended to document this deployment, not to serve as standardized cross-model benchmarks.
Fixed 4,096-token generation
Same prompt and generation setup for all configurations:
| Configuration | Median throughput |
|---|---|
| Swift FP8, target model only | 50.45 tok/s |
| Swift FP8 + native NEXTN/MTP | 103.38 tok/s |
| Swift FP8 + DFlash2, 8 draft tokens | 132.62 tok/s |
Measured DFlash2 runs:
131.24 tok/s
132.65 tok/s
132.62 tok/s
median: 132.62 tok/s
Compared with target-only generation, DFlash2 produced approximately 2.63x higher output throughput in this test.
Compared with native NEXTN/MTP, DFlash2 was approximately 28% faster.
The DFlash2 draft model is external and is not included in this repository.
Reasoning-heavy agent test
A separate local A/B test used the same diagnostic prompt, sampling parameters, seed, and reasoning setting for both systems.
The prompt asked the model to diagnose an intermittently slow OpenAI-compatible inference deployment with high GPU utilization, large KV cache, speculative decoding, variable context sizes and concurrency-sensitive latency.
| Metric | Previous Qwen FP8 production | Swift FP8 + DFlash2 |
|---|---|---|
| Wall time | 193.48 s | 139.63 s |
| Prompt tokens | 229 | 229 |
| Reasoning tokens | 14,838 | 10,716 |
| Completion tokens | 22,848 | 16,170 |
| Finish reason | stop | stop |
| Effective completion throughput | 118.09 tok/s | 115.80 tok/s |
Observed change:
- wall-clock time: approximately -27.8%
- reasoning tokens: approximately -27.8%
- completion tokens: approximately -29.2%
The main benefit in this test was not higher raw per-token throughput. Swift reached a similarly useful diagnostic answer with substantially fewer reasoning and completion tokens.
This is a local workload test and should not be interpreted as a standardized quality benchmark.
SGLang usage
Basic serving
python -m sglang.launch_server \
--model-path /path/to/Swift-Qwen3.8-27B-FP8 \
--served-model-name Swift-Qwen3.8-27B-FP8 \
--host 0.0.0.0 \
--port 30000 \
--attention-backend flashinfer \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
Native NEXTN / MTP speculative decoding
The retained native MTP head can be used with SGLang:
python -m sglang.launch_server \
--model-path /path/to/Swift-Qwen3.8-27B-FP8 \
--served-model-name Swift-Qwen3.8-27B-FP8 \
--host 0.0.0.0 \
--port 30000 \
--attention-backend flashinfer \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4
In the validated SGLang build, NEXTN is internally represented through the EAGLE speculative-decoding path.
DFlash2 speculative decoding
Best local throughput was obtained with a compatible external DFlash2 draft checkpoint:
python -m sglang.launch_server \
--model-path /path/to/Swift-Qwen3.8-27B-FP8 \
--served-model-name Swift-Qwen3.8-27B-FP8 \
--host 0.0.0.0 \
--port 30000 \
--attention-backend flashinfer \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-algorithm DFLASH \
--speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \
--speculative-num-draft-tokens 8
The DFlash2 weights are not redistributed in this repository.
Because speculative decoding verifies proposed tokens against the target model, the external draft model affects acceptance rate and speed rather than replacing the target model's token distribution.
Notes
This repository contains the quantized target checkpoint only.
It does not include:
- a DFlash2 draft checkpoint
- the original BF16 Swift checkpoint
- SGLang runtime binaries or containers
Performance depends heavily on GPU architecture, SGLang version, attention backend, context length, concurrency, KV-cache configuration and speculative decoding parameters.
License
This checkpoint is derived from:
and follows the Swift Open License v1.0 applicable to the upstream model.
Please refer to the upstream repository and its license text for the authoritative licensing terms.
No additional rights to the upstream model are granted by this repository.
Attribution
Original model:
- UkisAI
ukisai/Swift-Qwen3.8-27b
FP8 conversion, validation and local performance measurements for this repository were performed independently by the repository maintainer.
Citation
For the underlying Swift model, please cite or reference the upstream project:
@misc{swift-qwen3.8-27b,
title = {Swift-Qwen3.8-27B},
author = {UkisAI},
year = {2026},
url = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
}