NeLF S2T ONNX — Flemish Dutch ASR
ONNX export of the NeLF Speech-to-Text PyTorch model. Full Flemish/Dutch ASR with dual verbatim + subtitle output, running via ONNX Runtime — no PyTorch needed.
Model Architecture
- Encoder: STFT → LogMel (80) → GlobalMVN → Conv2dSubsampling6 → 12× Conformer (512-dim, 8 heads)
- Subtitle Encoder: 6× Transformer (cascaded from verbatim encoder)
- Decoders: Dual transformer decoders with swapped cross-attention (verbatim primary / subtitle primary)
- CTC head: Linear projection to 5000 tokens
- Params: 180M
Files
| File | Size | Description |
|---|---|---|
encoder.onnx | 348 MB | Audio waveform → verbatim encoder features (B, T, 512) |
encoder_sub.onnx | 83 MB | Verbatim features → subtitle encoder features |
ctc.onnx | 9.8 MB | Encoder features → CTC logits (B, T, 5000) |
decoder_step.onnx | 150 MB | Autoregressive decoder step (verbatim) |
decoder_sub_step.onnx | 150 MB | Autoregressive decoder step (subtitle) |
bpe.model | 314 KB | SentencePiece BPE tokenizer |
tokens.txt | 42 KB | Token vocabulary (5000 tokens) |
Usage
ONNX Runtime (Python)
import onnxruntime as ort
import numpy as np
import soundfile as sf
from sentencepiece import SentencePieceProcessor
# Load models
enc = ort.InferenceSession("encoder.onnx")
enc_sub = ort.InferenceSession("encoder_sub.onnx")
ctc = ort.InferenceSession("ctc.onnx")
# Load tokenizer
tokenizer = SentencePieceProcessor()
tokenizer.Load("bpe.model")
with open("tokens.txt") as f:
token_list = [line.strip() for line in f]
# Transcribe
audio, sr = sf.read("speech.wav")
audio = audio.mean(axis=1) if audio.ndim > 1 else audio
wav = audio[np.newaxis, :].astype(np.float32)
wav_len = np.array([wav.shape[1]], dtype=np.int64)
enc_out, enc_len = enc.run(None, {"wav": wav, "wav_len": wav_len})
logits = ctc.run(None, {"enc_out": enc_out})[0]
# Greedy CTC decode
preds = logits[0].argmax(axis=-1)
tokens = []
prev = -1
for p in preds[:int(enc_len[0])]:
if p != prev and p != 0:
tokens.append(int(p))
prev = int(p)
text = tokenizer.DecodePieces([token_list[i] for i in tokens])
print(text)
Autoregressive Decoding
dec = ort.InferenceSession("decoder_step.onnx")
tokens = [1] # <sos/eos>
for _ in range(225):
t = np.array([tokens], dtype=np.int64)
logits = dec.run(None, {
"tokens": t,
"encoder_out": enc_out,
"enc_len": enc_len,
"sub_out": enc_out, # or encoder_sub output for subtitle decoder
"sub_len": enc_len,
})[0]
next_t = int(logits[0, -1].argmax())
if next_t == 1:
break
tokens.append(next_t)
Input / Output Shapes
| Model | Inputs | Outputs |
|---|---|---|
encoder.onnx | wav: (B, T_wav), wav_len: (B,) | enc_out: (B, T_enc, 512), enc_len: (B,) |
encoder_sub.onnx | enc_in: (B, T, 512), enc_len: (B,) | sub_out: (B, T, 512), sub_len: (B,) |
ctc.onnx | enc_out: (B, T, 512) | logits: (B, T, 5000) |
decoder_step.onnx | tokens: (B, L), encoder_out: (B, T, 512), enc_len: (B,), sub_out: (B, T, 512), sub_len: (B,) | logits: (B, L, 5000) |
All axes are dynamic (variable batch size, time, sequence length).
Training Details
Trained on 300 hours of verbatim Flemish CGN (3× noise augmented), 700 hours of Netherlands Dutch CGN, and 14,000 hours of weakly-supervised Flemish broadcast subtitles. See the original model card for full details.
License
CC-BY-NC-4.0 (same as the original NeLF model).