mlx-community/GOT-OCR2_0-bf16
bf16 MLX conversion of stepfun-ai/GOT-OCR2_0, the 560M-parameter
OCR-2.0 model. Converted with mlx-vlm 0.6.14 (mlx 0.32.0) for inference on
Apple Silicon.
Fidelity was measured against the unquantized bf16 source and against the reference PyTorch implementation, not assumed. Every number below comes from a run recorded in this card.
Requires
mlx-vlmwith GOT-OCR 2.0 support, which is currently open as Blaizzy/mlx-vlm#1908. The PR is not merged yet, so a releasedmlx-vlmcannot load this repo.
python -m mlx_vlm generate \
--model mlx-community/GOT-OCR2_0-bf16 \
--image document.png \
--prompt "OCR: " \
--max-tokens 1024
GOT is not a chat model. It takes two instructions: OCR: for plain text and
OCR with format: for structured output (tables, formulas, sheet music). Any
other prompt is out of distribution.
Conversion
| Precision | bfloat16 |
| Bits/weight | 16.0 |
| Size on disk | 1.12 GB |
| Parameters | 560.5M (716.0M stored, tied lm_head dropped) |
No quantization. This is the reference the two quantized variants were measured against, and the one to use when fidelity matters more than memory.
Against the reference PyTorch implementation
One forward pass, same image and same prompt (the MPT conversation the model was trained with), comparing the final-position logits over all 151,860 classes.
| Comparison | max abs error | cosine | argmax |
|---|---|---|---|
| MLX bf16 vs torch fp32 | 0.47469 | 0.99979109 | same, top-5 identical |
| MLX fp32 vs torch fp32 | 0.000130 | 1.0000000000 | same |
The second row is the one that matters: the source weights are bf16, so running the MLX port in fp32 loses nothing, and the residual is porting error alone. The first row's larger figure is bf16 rounding, not a defect.
Task level, against ground truth
Six documents rendered locally with exactly known text (invoice, lab report,
shipping label, receipt, spec table, rotated receipt), transcribed with OCR:
and scored directly. This sidesteps using bf16 as the reference at all.
| Variant | field | content | numeric | CER vs bf16 | tok/s | peak GB |
|---|---|---|---|---|---|---|
| bf16 | 0.8684 | 0.9605 | 0.9720 | 0 (ref) | 138.3 | 2.50 |
| 8-bit | 0.8684 | 0.9605 | 0.9720 | 0.0000 | 210.8 | 2.06 |
| 4-bit | 0.8947 | 0.9474 | 0.9623 | 0.0116 | 272.9 | 1.83 |
field counts required strings present exactly, content ignores markup and
whitespace, numeric counts ground-truth numbers recovered.
The 8-bit output is byte-identical to bf16 on all six documents, down to the same fields missed.
Two caveats a reader should have:
- The 4-bit
fieldscore is higher than bf16 and that is not an improvement. The denominator is 38 strings, so one string is 0.026. 4-bit happens to get the invoice fully right and then loses12.00on the receipt.contentandnumeric, which are less format-sensitive, both move the other way. Treat the three variants as separated by one or two strings, not byfield. shipping_labelloses the same three fields in every variant, bf16 included (1Z-994-AX-77310582,Halberd Components,3011AB). That is a limit of the source model on that document, not a quantization effect.
Throughput measured on an M-series Mac during the same run, single image, generation only.
What was not measured
- Only the plain 1024x1024 single-crop path. GOT's fine-grained mode (region by box or by colour), its multi-crop path for dense pages, and its multi-page mode are not exercised by anything above.
OCR with format:was not scored. The accuracy table usesOCR:only. Format mode was smoke-tested and runs, but no ground-truth scoring was done for tables, formulas or sheet music.- No perplexity, top-1 agreement or KL. GOT only emits transcriptions, so generic passages give meaningless perplexity. The ground-truth OCR scoring above replaces that layer rather than supplementing it.
- Six synthetic documents, one language. They are rendered, not photographed, and English only. Real scans, handwriting and the multilingual claims of the source are untested here.
- No accuracy measurement at long context. Everything ran well under the 32k window.
Related
Source model: stepfun-ai/GOT-OCR2_0 (paper).