whisper-edge
Whisper encoders compiled for edge accelerators, plus the CPU decoders they pair with.
Consumed by openvoicestream's
voxedge.backends.whisper backend; server/core/model_downloader.py fetches from this
repo. Measurements across five boards, and the reasoning behind each choice, are in
docs/perf/whisper-cross-device-20260827.md there.
Layout
| Path | For |
|---|---|
encoder/hailo/tiny-whisper-encoder-10s_15dB.hef | Hailo-8, tiny, 10 s window |
encoder/hailo/base-whisper-encoder-5s.hef | Hailo-8, base, 5 s window |
encoder/rk/whisper_encoder_base_10s.rknn | RK3588 / RK3576, base, 10 s window |
encoder/rk/whisper_encoder_base_20s.rknn | RK3588, base, 20 s window |
encoder/jetson/enc_base_30s.onnx | Jetson — input for an on-device TensorRT build |
decoder/base/ | Shared by every base encoder above |
decoder/tiny/ | Hailo tiny only |
mel_80_filters.txt, vocab_en.txt, vocab_zh.txt | Shared front end and vocabulary |
Notes
The decoder runs on the CPU deliberately. Neither vendor's NPU decoder has a KV cache — Hailo compiles a fixed 32-token sequence, Rockchip a 12-slot sliding window — so both recompute the whole sequence every autoregressive step. Running the encoder on the accelerator and the decoder as a CPU ONNX graph measured both faster and more accurate on every board: RK3588 English long-form went 10.44% → 7.58% WER while RTF went 0.149 → 0.061.
tiny and base decoders are not interchangeable (4 layers / d384 against 6 / d512). Crossing them produces fluent nonsense rather than an error.
Build the Jetson engine with --bf16, not --fp16. The fp16 build of that encoder
graph scores cosine 0.826 against onnxruntime and fails silently — the decoder emits
fluent text that drifts off-topic. bf16 scores 0.9996 and matches fp32's error rates.
Passing --fp16 --bf16 together yields an engine bit-identical to the pure fp16 one.
The window is fixed at compile time. It must match the seconds in the filename; rknn-lite does not validate it, and a mismatch reinterprets the buffer and returns plausible nonsense rather than an error.
Whisper here is for English. Chinese CER measured 35–56% across every board, which is Whisper base/tiny's own ceiling rather than anything a faster accelerator fixes. Use Paraformer, SenseVoice or Qwen3-ASR for Chinese.
Provenance
The Hailo HEFs come from Hailo's own edge_whisper example assets and the RKNN encoders
from Rockchip's model zoo conversion flow; the ONNX decoders are optimum exports of
OpenAI Whisper. Original licences apply to each.