Cohere Transcribe 2B — CoreML INT8 (W8A16) for Apple Silicon & iOS
This repository provides W8A16 Linear Symmetric Quantized CoreML (.mlpackage) models for Cohere Transcribe 2B (c4ai-transcribe-2b), optimized for high-performance speech-to-text on Apple Silicon (M1/M2/M3/M4/M5) and mobile Apple devices.
Quantized using Apple's official coremltools.optimize.coreml pipeline with OpLinearQuantizerConfig(mode='linear_symmetric', dtype='int8').
Performance Highlights (Apple M5 Silicon)
Benchmarked on the full 15.08-minute (904.93s) JFK Inaugural Address:
| Metric | Lossless FP16 Baseline | W8A16 INT8 CoreML Engine | Advantage |
|---|---|---|---|
| Total Audio Duration | 904.93 s (15.08 min) | 904.93 s (15.08 min) | Identical |
| Total Compute Time | 15.913 s | 14.837 s | -1.08s faster |
| Real-Time Factor (RTF) | 0.0176 | 0.0164 | Lower is better |
| Throughput Speedup | 56.9x Real-Time | 61.0x Real-Time | +4.1x RT throughput |
| Peak Chunk Speedup | ~61x RT | 73.8x Real-Time (270 ms / 20s) | Instantaneous |
| Decoder Weight Size | 258 MB | 129 MB | 50% reduction |
| Accuracy | 1411 / 1411 words (100%) | 1437 words | Verbatim fidelity |
Architecture Partitioning
- Conformer Encoder (
cohere_encoder.mlpackage): Runs natively on the Apple Neural Engine (ANE), processing 20s of audio in ~190 ms while consuming 0% of the GPU. - Transformer Prefill (
cohere_decoder_prefill.mlpackage): INT8 weights (145 MB), computes prompt representation in ~30 ms on GPU. - Autoregressive Decoder (
cohere_decoder_decode.mlpackage): INT8 weights (129 MB) utilizing in-place Metal buffer mutation viaMLState. Autoregressive single-step token latency drops from $1.68\text{ ms} \rightarrow 0.84\text{ ms}$ weight memory read time.
Quick Setup with Voxtype
To install and compile these INT8 CoreML models into Voxtype:
cd ~/voxtype
python3 scripts/setup_cohere_coreml.py --repo-id ctyau/cohere-transcribe-coreml-int8
This will automatically:
- Download the
.mlpackagebundles. - Compile them into native
.mlmodelcbundles for your specific Mac. - Install them into
~/Library/Application Support/voxtype/models/cohere-transcribe-fp16/compiled_coreml.
Model Card & License
- Base Model: CohereForAI/c4ai-transcribe-2b
- License: CC-BY-NC-4.0 (inherits base model license)