GLiNER2.5-Decide 340M MLX INT8 (mixed)
MLX conversion of fastino/GLiNER2.5-Decide
for native inference on Apple Silicon with speech-swift.
Given text and a label schema, it returns a probability for every label
(single-label classification) or spans of the original text (entity extraction).
It does not generate text.
Converted from the upstream checkpoint at revision 7ee5da4c2415e32259bcdc0b1a7367c32ce8d6f6.
Model
| Architecture | DeBERTa-v3-large encoder, GLiNER2 span and classification heads |
| Parameters | 340M (upstream figure) |
| Precision | INT8 (mixed): 8-bit affine weights (group 64) for encoder matrices and token embeddings; float16 heads, norms, relative embeddings and activations |
| Weights | 567 MB, SHA-256 61e91848b390fbf23dcbe76419d835644f76afefe554de3e6d1789d5e1e4f282 |
| Context | 512 encoded tokens (schema and text together) |
| Language | English |
Files
| File | Size | Contents |
|---|---|---|
LICENSE | 11.4 kB | Apache-2.0 (model weights) |
LICENSE-gliner2-mlx | 1.1 kB | MIT (reference conversion code) |
LICENSE-mlx-reference | 1.1 kB | MIT (MLX affine quantization equations used by the INT8 converter) |
UPSTREAM_README.md | 16.9 kB | Original Fastino model card |
config.json | 0.6 kB | GLiNER2 head configuration |
encoder_config/config.json | 0.9 kB | DeBERTa-v3-large encoder configuration |
export.json | 10.2 kB | Source revision, format and weight hash |
quantization.json | 9.3 kB | Quantized tensor list, bits and group size |
special_tokens_map.json | 2.4 kB | Special tokens |
tokenizer.json | 8.3 MB | Unigram tokenizer with GLiNER schema tokens |
tokenizer_config.json | 3.4 kB | Tokenizer settings |
weights.safetensors | 567.0 MB | Encoder and span/classification head weights |
Performance
Apple M5 Pro, idle machine, one process per variant, model loaded once. Median of full requests including tokenization: 16 routing cases with six labels, 8 entity cases with two labels, five timed calls each after five warmups.
| Routing (median) | Extraction (median) | Peak process memory |
|---|---|---|
| 7.6 ms | 8.9 ms | 0.85 GB |
Fidelity to the upstream PyTorch model on 24 reference cases: identical labels, spans and offsets; largest confidence difference 0.0060 (gate 0.02).
Usage
Swift (speech-swift GLiNER module):
import GLiNER
let model = try await GLiNER.fromPretrained(variant: .int8)
let choices = try model.classify(
"Remind me to call Dad at six PM.",
labels: ["create_reminder", "send_message", "other"]
)
let spans = try model.extractEntities(
"Remind me to call Dad at six PM.",
labels: ["person", "time"]
)
CLI:
speech gliner classify "Remind me to call Dad at six PM." --variant int8 \
--labels create_reminder,send_message,other
speech gliner extract "Remind me to call Dad at six PM." --variant int8 \
--labels person,time --json
Probabilities are model scores, not guarantees. Entity spans are mentions: dates and times are returned as text for your code to normalize. Offsets are UTF-16.
Source and licenses
- Model:
fastino/GLiNER2.5-Decide, Apache-2.0 (LICENSE). The weights are converted to MLX layout without retraining; the original card is kept asUPSTREAM_README.md. - Conversion follows gliner2-mlx
by Andrew Chen Wang, MIT (
LICENSE-gliner2-mlx). - GLiNER2 paper: arXiv:2507.18546.