Cohere Transcribe Arabic for Intel NPU
OpenVINO port of CohereLabs/cohere-transcribe-arabic-07-2026, a 2 billion parameter Arabic speech recognition model, packaged to run fully on an Intel NPU. Tested on Core Ultra 7 255H (Arrow Lake) with intel level zero NPU driver 1.33 and OpenVINO 2026.4.
Contents
- OpenVINO IR for the Conformer encoder, FP16 and INT8
- OpenVINO IR for a stateful one token decoder with on device KV cache, FP16 and INT8
- OpenVINO IR for the cross attention K/V precompute used once per audio chunk, FP16 and INT8
- The original cache less decoder IR, FP16 and INT8, kept for comparison
- Tokenizer, mel filterbank, output projection and runtime prompt constants
- run_local.py, a small NumPy based runner that needs no torch at inference time
- WER benchmarks against the original model, see the benchmarks folder
File sizes are roughly: encoder 3.8 GB in FP16 and 1.9 GB in INT8, cache less decoder 307 MB and 154 MB, KV step decoder 271 MB and 136 MB, cross K/V precompute 36 MB and 18 MB.
Accuracy On 150 FLEURS Arabic clips, 27.7 minutes of audio, the INT8 KV cache pipeline on NPU scores 9.03 percent WER against 8.96 percent for the original model on GPU. The paired difference is plus 0.07 points with a 95 percent interval from minus 0.10 to plus 0.24. 96 percent of transcripts are identical. Full numbers with per clip references and hypotheses are in benchmarks/wer_paired_150.json, and the raw Kaggle baseline output is in benchmarks/wer_hf_baseline.json. Notes on the absolute WER level are in benchmarks/notes.md.
Speed on Core Ultra 7 255H NPU, 8.5 second Arabic clip, 66 tokens
- encoder plus cross attention precompute: about 1.1 s
- decoder: about 1.5 s, around 23 ms per token
- end to end: about 3.5 times real time
Quickstart
python -m pip install -r requirements.txt
python run_local.py -b . -d NPU -c kv -p int8 -l ar examples/sample1.wav
Flags used above: bundle directory, device, cache mode (kv for the stateful
decoder, none for the original full recompute decoder), precision (fp16,
int8, int4 for the step decoder), language (ar or en). Other devices work as
well, for example -d GPU. The INT8 encoder does not compile on the CPU
plugin, use FP16 there. Mixed placement is possible through the longer device
options shown by the built in help. The first run on NPU compiles the graphs
and caches the blobs, later runs start fast.
How it was built The PyTorch checkpoint was converted to OpenVINO IR. The decoder was split into a cross attention K/V graph evaluated once per chunk and a stateful one token step graph whose self and cross attention caches live in on device states, so each generated token costs one token of work instead of a full 512 token recompute. INT8 variants use NNCF weight compression. The encoder mask constants were changed from negative infinity to a finite value so real attention masks can be used; with the original constants, padded frames made the encoder output NaN.
License Apache 2.0, same as the base model. Please cite the base model card (CohereLabs/cohere-transcribe-arabic-07-2026) if you use this port.