Hy-MT2-30B-A3B NVFP4 Experts GGUF
An expert-only NVFP4 derivative of tencent/Hy-MT2-30B-A3B for native
English-Persian and Persian-English translation on NVIDIA Blackwell GPUs.
This is not a blanket 4-bit conversion. The 128 routed experts in layers 1-47 are NVFP4; attention, embeddings, routers, shared experts, layer 0, and the language-model head remain BF16. The routed experts account for 94.4198% of the 30,064,719,872 source parameters.
Provenance
- Source:
tencent/Hy-MT2-30B-A3B - Source revision:
d3ead4dba61c09aac60a261a96ad1df3e705febb - Quantizer: NVIDIA ModelOpt 0.45.0 at
ec87a82927d003986d44fb7f4fa8b3d10c31b095 - Runtime/converter: llama.cpp at
ebb546b7e961bd46fd9ed0387ffd14ca86b6fe1b - Calibration: 240 English/Persian Wikipedia samples, 65,487 tokens, collected on 2026-08-11; benchmark sources and references were excluded
- Artifact size: 19,353,433,056 bytes
- SHA-256:
3f5edeb940b56ad851cce84cc55de7812d97dc80aeb4a6d96aa20f044bec891f
ModelOpt fake-quant validation on eight held-out calibration rows produced 0.979527 prefill-logit cosine similarity and 0.973255 first-decode-logit cosine similarity, with 8/8 top-1 agreement in both cases.
Validation
The native GGUF was tested on an RTX PRO 6000 Blackwell Server Edition with a 64-row balanced development set (32 translations in each direction). References are synthetic development evidence, not an independent human test set.
| Runtime | Format | Output tok/s | en-fa chrF++ | fa-en chrF++ |
|---|---|---|---|---|
| vLLM | BF16 source | 859.40 | 56.86 | 66.00 |
| llama.cpp | NVFP4 GGUF | 413.92 | 56.84 | 65.49 |
Both runs used batch/concurrency 64 and a maximum of 128 generated tokens. The NVFP4 run produced no empty translations and no exact source copies. Its output agreement with BF16 was 88.01 chrF++ for English-to-Persian and 93.66 chrF++ for Persian-to-English.
RTX 5080 Laptop (Stallion)
The same GGUF and llama.cpp revision were also validated on a 16 GB RTX 5080 Laptop GPU at an 85 W power limit. With 38 layers offloaded to CUDA and the remaining weights in system RAM, a single 8K-context slot used 15,144 MiB VRAM and generated at 53.9 tok/s server-side (51.9 tok/s end-to-end).
An eight-slot throughput run used 8K total context (1,024 tokens per slot), 64 requests, and a 128-token output cap:
| Concurrency | Output tok/s | Total tok/s | Requests/s |
|---|---|---|---|
| 8 | 85.63 | 231.87 | 1.69 |
That run produced no empty outputs or exact source copies. Development chrF++ was 56.17 for English-to-Persian and 65.80 for Persian-to-English.
llama.cpp
Use a CUDA build that includes SM120 and the NVFP4 MMQ kernel. The exact tested llama.cpp revision is listed above.
cmake -S . -B build-sm120 \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build-sm120 --config Release -j --target llama-cli llama-server
build-sm120/bin/llama-cli \
-m Hy-MT2-30B-A3B-NVFP4-Experts.gguf \
-ngl 99 -c 4096 -fa on \
-n 256 --temp 0
The amount of GPU offload is hardware- and workload-dependent. Leave headroom for the KV cache and other processes instead of assuming the complete 19.35 GB file fits in VRAM.
The tested 16 GB / 8K Stallion configuration used -ngl 38 -c 8192 -np 1 for
single-request serving. For the aggregate batch-8 result it used
-ngl 38 -c 8192 -np 8; llama.cpp therefore assigned 1,024 context tokens to
each parallel slot.
Important limitation
The intermediate ModelOpt safetensors pass fake-quant logit checks, but their current native and emulated vLLM deployment paths generated invalid text during this build. Those files are quarantined and are not the supported artifact. The GGUF listed by the exact checksum above is the validated runtime artifact.
The development set was used to validate conversion fidelity. It must not be reported as a fresh held-out measure of general translation quality.
Licensing and attribution
This model distribution is licensed under the Apache License, Version 2.0. See LICENSE. Existing third-party copyright, license, and attribution notices remain applicable.
- Upstream: tencent/Hy-MT2-30B-A3B; declared license:
apache-2.0.- Retained notice: LICENSE.txt.