Swift-Qwen3.8-27b-oQ4e-mtp
Model architecture
A 27B-class dense multimodal model stored in MLX format. It contains a text backbone, a vision encoder, and one multi-token prediction (MTP) layer.
| Component | Structure |
|---|---|
| Text backbone | 64 layers; hidden size 5,120; feed-forward size 17,408 |
| Attention layout | 48 linear-attention layers and 16 full-attention layers, with full attention every fourth layer |
| Full attention | 24 query heads, 4 key/value heads, head dimension 256 |
| Vocabulary | 248,320 tokens |
| Configured context limit | 262,144 tokens; usable length depends on runtime settings and available memory |
| Vision encoder | 27 layers; hidden size 1,152; 16 attention heads; 16 × 16 image patches |
| Vision-to-text connection | Vision features are projected to the text hidden size of 5,120 |
| MTP | One additional prediction layer with attention and feed-forward projections |
Weight precision
The oQ4e checkpoint uses mixed precision rather than uniform 4-bit weights:
- The default quantization is 4-bit affine, with 64 values per group.
- 187 modules have explicit 5-bit overrides in
config.json. - The MTP layer's seven large attention and feed-forward matrices use 4-bit weights.
- The MTP fusion matrix (
mtp.fc) and normalization weights remain BF16. The fusion matrix maps 10,240 input features to 5,120 output features. - The MTP quantization scales and offsets are stored in BF16.
This checkpoint retains BF16 floating-point weights in the MTP fusion and normalization layers alongside its quantized matrices. Exact per-module settings are recorded in config.json.
Source and license
Source revision: 54e66d6c81439bd4fda5ef9a690fa571e3b0d272.
Original model by UkisAI. This conversion does not change the upstream Swift Open License v1.0 terms. Consult the source model license and access information.