Unlimited-OCR — Community FP8-Dynamic
Unofficial community quantization — not a Baidu release.
FP8 W8A8 build of baidu/Unlimited-OCR (revision
07dea832), the 3.34B
DeepSeek-V2-style MoE OCR model ("one-shot long-horizon parsing"), validated against a BF16
baseline of the published checkpoint. All credit for the model belongs to Baidu; this repo changes
only the numeric precision of the text-decoder weights. Checkpoint size drops from 6.7 GB to
3.9 GB.
Pick this variant for Ada/Hopper/Blackwell GPUs where you want near-lossless behavior with native FP8 execution. The companion NVFP4 build — calibrated, unlike prior community 4-bit builds — shrinks the footprint further for memory-constrained Blackwell deployments.
What is quantized
Scheme: FP8_DYNAMIC via llm-compressor —
per-channel FP8 weights, dynamic per-token FP8 activations, data-free. Only the DeepSeek-V2 MoE
text-decoder linears are quantized (2196 modules: 2112 routed-expert, 33 shared-expert/dense-MLP,
48 attention, 3 dense-layer). Everything the vision path depends on stays BF16:
- SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
embed_tokens/lm_head- MoE router gates and all norms
Validation
Greedy OCR on the fixture set (vLLM 0.26.0, SM120) is character-identical to the BF16
baseline — CER 0.0 on every fixture, grounding-box coordinates included. The same parity holds
under transformers + compressed-tensors.
| BF16 | FP8-Dynamic (this repo) | |
|---|---|---|
| Mean CER vs BF16 | — | 0.0000 |
| invoice / memo / table CER | — | 0.0 / 0.0 / 0.0 |
| Decode throughput (tok/s, greedy) | 45.3 | 39.4 |
| Checkpoint size | 6.7 GB | 3.9 GB |
Raw per-fixture transcripts ship under evidence/, produced by the
validation harness.
Note: the shipped quantization_config.ignore uses prefix-agnostic regex patterns — required for
vLLM to correctly skip the vision tower for quantization compatibility. Do not replace it with expanded
module names.
Usage
import torch
from transformers import AutoModel, AutoTokenizer
repo = "shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
image_file="document.png", output_path="./out",
base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)
About
Quantized by Matt Busi at ShadowRock. Raw evaluation outputs ship under
evidence/.