Sidon — Core ML
Core ML conversions of sarulab-speech/sidon-v0.1, a speech restoration model, rebuilt to run on an iPhone's Neural Engine. Used by Excerpt, an iOS audiobook player, to lift noise and muffling out of narration on the device.
16 kHz in, 48 kHz out, duration preserved. The weights are upstream's, unchanged — nothing here is fine-tuned. What changed is the graph around them, so that it compiles for the Neural Engine.
Contents
| File | Precision | Size | Fixed input |
|---|---|---|---|
sidon_encoder_ane.mlpackage | fp16 | 468 MB | feats [1, 300, 160] -> hidden [1, 300, 1024] |
sidon_decoder_ane.mlpackage | fp16 | 101 MB | feat [1, 1024, 68] -> [1, 1, 65280] |
sidon_fbank.bin | float32 | 84 KB | Front-end constants: 400-tap Povey window and 257 x 80 mel filters |
.mlpackage files are uncompiled. Compile them on the device with
MLModel.compileModel(at:) and cache the result; the first compile can take a
minute or more. Load both with MLComputeUnits.cpuAndNeuralEngine.
How this differs from upstream
The feature extractor is not in the model. Upstream feeds raw audio through
SeamlessM4TFeatureExtractor (from facebook/w2v-bert-2.0) in Python. Here the
encoder takes the 160-dimensional stacked fbank features directly, and
sidon_fbank.bin carries that extractor's exact window and mel filters so the
features can be computed natively. Recomputing them from a formula instead shifts
every frame. The extractor normalises per utterance, so compute features once over
a long stretch of audio and slice the features, never the audio.
The encoder runs 300 frames (6 s) per call. Upstream has no fixed length. The Neural Engine compiler fails at 600 frames — it runs out of memory expanding the relative position tables, and on a phone it is killed. 300 frames compiles and runs 99.3 % on the Neural Engine. The conformer uses long-range context, so shorter windows cost accuracy: against 12 s windows this measures about 2 dB lower SNR, which was inaudible in blind listening tests. The encoder's dictionary output is also rewired to a plain tensor, since Core ML has no dictionary op; that surgery is checked against the original to 1e-5.
The decoder is rebuilt, and its transposed convolutions are rewritten.
Upstream's DAC decoder is a frozen TorchScript module whose scripted Snake
activations cannot be converted. It is reconstructed in Python from the repo
config with the weights transplanted in, checked bit-exact against the original.
Its ConvTranspose1d layers — which the Neural Engine computes wrongly, dropping
SNR by 53 dB at a single layer — are replaced with an equivalent sub-pixel form:
two convolutions whose outputs interleave. Because every layer has
kernel == 2 * stride, this is exact, not an approximation (max|d| 0.0 in
PyTorch). It is also 1.65x faster on the iPhone GPU.
The decoder runs 68 frames per call. The Neural Engine caps every tensor dimension at 65536, and the decoder's output is 960 samples per frame, so 68 frames is the most that fits. Run it in overlapping windows with a 12-frame halo on each side and keep only the middle; that reconstructs a single long call to 120 dB. Every window must hold real frames — slide the last one back rather than zero-padding it.
What it costs
Against the same models on the GPU, the Neural Engine output measures 1.7 dB lower, from fp16 hidden states and the shorter encoder window. The difference sits above 16 kHz, 52 dB down; across the speech band it is within 0.4 dB. In a blind, shuffled listening test, PyTorch, all-GPU, GPU + Neural Engine and all-Neural Engine renders could not be told apart.
On an iPhone, warm, this runs at 11.2x realtime, against 4.8x on the CPU alone. A backgrounded iOS app cannot use the GPU, so for long audio that is the comparison that matters.
What it does
Sidon re-synthesises speech from w2v-BERT semantic features rather than repairing the waveform. Running it on clean audio changes it about as much as running it on damaged audio. Expect cleaner, clearer narration in the same voice, not a sample-accurate restoration of the original recording.
Licence and attribution
MIT, inherited from the upstream model — see LICENSE. The model, its training
and its weights are the work of the Sidon authors (Wataru Nakata, Yuki Saito);
this repository contributes only the Core ML conversion.