NLLB-200 Distilled 600M — INT8 CPU Quantized (PyTorch Dynamic)
[!NOTE] 🌐 Live Web Demo: Try 100% offline in your browser at
huggingface.co/spaces/rudrakshrakeshzodage/nllb-200-onnx-browser-demo
[!NOTE] This model is part of a suite of optimized/quantized versions of the base model. Other variants in this direction:
- FP32 (Full Precision / Base):
facebook/nllb-200-distilled-600M- FP16 (Half Precision):
facebook/nllb-200-distilled-600M- INT8 (Dynamic Quantization - CPU):
rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu(Current)- NF4 (4-bit GPU Quantization):
rudrakshrakeshzodage/nllb-200-distilled-600M-nf4-4bit-gpu- ONNX INT8 (Mobile & Edge Deployment):
rudrakshrakeshzodage/nllb-200-distilled-600M-onnx-int8(Current)
Base model: facebook/nllb-200-distilled-600M
Quantization: PyTorch INT8 Dynamic (torch.quantization.quantize_dynamic)
Languages: 200 (all NLLB-200 supported languages)
💾 Model Size Reduction Comparison (FP32 vs FP16 vs INT8 vs NF4)
📈 200+ Language Performance Benchmark Chart
Quantization Architecture Details
- Target Layers:
torch.nn.Linearinside Encoder & Decoder Self-Attention / FFN modules. - Weight Precision:
qint8(8-bit signed integers). - Activation Quantization: Calculated dynamically at inference runtime per batch.
- No GPU Needed: Optimized for x86 CPU SIMD / AVX2 / AVX-512 instructions.
📊 Model Performance & Benchmarks
| Metric | INT8 Dynamic (CPU) | FP16 Base (CPU) | Speedup / Savings |
|---|---|---|---|
| RAM Footprint | ~800 MB | ~2,400 MB | 66.7% RAM reduction |
| Avg Latency (Per Lang) | ~1,097 ms | ~3,950 ms | 3.6× Faster |
| Throughput | 55 languages / min | ~15 languages / min | 3.6× Higher |
| 200-Lang Benchmark Run | ~3.7 minutes | ~13.3 minutes | 9.6 min saved |
| Loop Anomalies | 0 ✅ | 1 (Central Kanuri) | Fully stable |
| BLEU Score (vs FP16) | ~42.0 | ~42.1 | < 0.1 BLEU loss |
💻 Hardware Requirements
- CPU: Intel Core i3 (10th Gen+), AMD Ryzen 3 (3000U+), Apple Silicon (M1+)
- RAM: 4 GB System RAM minimum (800 MB active model memory)
- GPU: None required (100% CPU inference)
🚀 Usage
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
# Load base architecture
model = AutoModelForSeq2SeqLM.from_pretrained(
"facebook/nllb-200-distilled-600M"
)
# Apply PyTorch INT8 Dynamic Quantization
model = torch.quantization.quantize_dynamic(
model, {torch.nn.Linear}, dtype=torch.qint8
)
tokenizer = AutoTokenizer.from_pretrained(
"YOUR_USERNAME/nllb-200-distilled-600M-int8-cpu"
)
# Translate English to Hindi (hin_Deva)
inputs = tokenizer("Hello world, how are you?", return_tensors="pt")
translated_tokens = model.generate(
**inputs,
forced_bos_token_id=tokenizer.lang_code_to_id["hin_Deva"],
max_length=100
)
print(tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0])
📜 Citation & Credits
Quantization research and benchmarking by RudrakshRakeshZodage.