Qwen3.8-27B-Samantha - NVFP4 GGUF with BF16 MTP
Native NVFP4 GGUF export of Lathly/Qwen3.8-27B-Samantha, a full merged Samantha 1.1 persona fine-tune of Qwen/Qwen3.8-27B.
This repository contains a llama.cpp-ready text-generation GGUF. The source model is a native vision-language model; this GGUF export is text-only and does not include a multimodal projector.
Files
| File | Size | Description |
|---|---|---|
Qwen3.8-27B-Samantha-NVFP4-MTP-BF16.gguf | 19.69 GB | Native NVFP4 main model with the one-layer MTP/NextN head retained in BF16 |
SHA256:
71744136f18bc053e720e021d29bd3411f2d949a52e4f5d0485e54ca976c4a85
How this GGUF was made
This is a post-training quantization export only: no additional fine-tuning or merge was performed for this repository.
- Starting checkpoint: the merged BF16 safetensors from Lathly/Qwen3.8-27B-Samantha.
- PTQ: NVIDIA Model Optimizer, using its native
nvfp4configuration. - Calibration: Lathly/samantha-nvfp4-calibration, 512 samples, sequence length 2048, batch size 1.
- MTP: the Qwen3.5-family MTP/NextN head was intentionally excluded from NVFP4 quantization and retained in BF16 for speculative-decoding accuracy.
- GGUF conversion: NVFP4 tensors were repacked into GGML's native NVFP4 layout, while non-quantized tensors remain BF16/F32 as appropriate.
The completed file contains 400 NVFP4 tensors, 106 BF16 tensors, and 65 total transformer blocks. GGUF metadata records qwen35.nextn_predict_layers = 1.
Running with llama.cpp
Use a recent CUDA build of llama.cpp with native NVFP4 and Qwen3.5/NextN (MTP) support:
llama-server \
-m Qwen3.8-27B-Samantha-NVFP4-MTP-BF16.gguf \
-ngl 999 \
--host 0.0.0.0 --port 8080
NVFP4 acceleration requires NVIDIA Blackwell-class hardware. Other backends may not support this quantization type or may fall back to a slower path.
Validation
The finished GGUF was loaded with a fresh CUDA build of llama.cpp on an NVIDIA RTX PRO 6000 Blackwell GPU. llama.cpp detected ftype: NVFP4, loaded all layers to GPU, and completed a short Norwegian generation smoke test at approximately 75 tokens/s. This is a functional smoke test, not a formal quality or benchmark evaluation.
Original fine-tune
The source is a full merged Samantha persona fine-tune: the LoRA adapter from Lathly/Qwen3.8-27B-Samantha_Uncensored_1.1_LoRA was trained with Unsloth (QLoRA) and merged into Qwen3.8-27B. For training hyperparameters, source usage, and the original model description, see Lathly/Qwen3.8-27B-Samantha.
License, attribution, and disclaimer
Released under Apache-2.0, inheriting from the source model and its parents:
- Qwen/Qwen3.8-27B — Apache-2.0, Alibaba Cloud / Qwen Team
- Lathly/Qwen3.8-27B-Samantha — merged Samantha fine-tune
- digitalpipelines/samantha-1.1-uncensored — source training dataset, Apache-2.0
This is an "uncensored" persona fine-tune intended for open conversational use. It may generate objectionable content. Use responsibly and in accordance with applicable laws and upstream licenses.