cohere-transcribe-03-2026 NVFP4 W4A16
This is an NVFP4 W4A16 (weights FP4, activations BF16) quantized version of CohereLabs/cohere-transcribe-03-2026, a 2B parameter Conformer-based encoder-decoder ASR model supporting 14 languages.
Quantization Details
| Metric | Value |
|---|---|
| Base Model | CohereLabs/cohere-transcribe-03-2026 |
| Quantization | NVFP4 W4A16 (weight-only) |
| Weight Format | NVIDIA FP4 (E2M1, block_size=16) |
| Activation Format | BF16 (unchanged) |
| Calibration | None (dynamic scales from weights) |
| Tool | NVIDIA ModelOpt v0.44.0 |
| Original Size | 3.9 GB (BF16) |
| Quantized Size | 1.5 GB (NVFP4) |
| Compression | ~2.6x |
Usage
With vLLM
vllm serve jeffpeng3/cohere-transcribe-03-2026-NVFP4 \
--quantization modelopt \
--dtype bfloat16
With Transformers
import torch
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
model_id = "jeffpeng3/cohere-transcribe-03-2026-NVFP4"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
# Transcribe an audio file
from transformers.audio_utils import load_audio
audio = load_audio("path/to/audio.wav", sampling_rate=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt", language="en").to(model.device)
inputs = inputs.to(dtype=model.dtype)
outputs = model.generate(**inputs, max_new_tokens=256)
text = processor.decode(outputs[0], skip_special_tokens=True)
print(text)
Supported Languages
- European: English, French, German, Italian, Spanish, Portuguese, Greek, Dutch, Polish
- APAC: Chinese (Mandarin), Japanese, Korean, Vietnamese
- MENA: Arabic
Model Architecture
- Encoder: Conformer (48 layers, d_model=1280)
- Decoder: Transformer (8 layers, hidden_size=1024)
- Parameters: 2B
- Tokenizer: SentencePiece (vocab_size=16384)
Output Structure
cohere-transcribe-03-2026-NVFP4/
├── config.json # Model config + quantization metadata
├── hf_quant_config.json # ModelOpt quantization config
├── model.safetensors # NVFP4 quantized weights (1.5 GB)
├── modeling_cohere_asr.py # Custom model code
├── processing_cohere_asr.py # Custom processor code
├── tokenizer.json / .model # SentencePiece tokenizer
└── preprocessor_config.json # Feature extractor config
vLLM Serving
For online inference with vLLM:
vllm serve jeffpeng3/cohere-transcribe-03-2026-NVFP4 \
--quantization modelopt \
--dtype bfloat16 \
--max-model-len 1024
License
Apache 2.0