WavCoch V8192 vocoder with causal and centered encoding

WavCoch is a causal waveform-to-cochleagram tokenizer by Greta Tuckute and Klemen Kotar.

Model Details

Usage

import torch
from transformers import AutoModel

wavcoch = AutoModel.from_pretrained(
    "TuKoResearch/WavCochCausalV8192-vocoder-centered",
    trust_remote_code=True,
)

# Default mode is identical to TuKoResearch/WavCochCausalV8192-vocoder.
causal_codes = wavcoch.quantize(waveform_tensor)
assert torch.equal(causal_codes, wavcoch.quantize(waveform_tensor, mode="causal"))

# Centered mode preserves the number of frames while geometrically centering
# the 1001-sample analysis window on token endpoints. It uses 421 left and
# 500 right zeros.
centered_codes = wavcoch.quantize(waveform_tensor, mode="centered")

codes = causal_codes
coch = wavcoch.decode(codes)
embeddings = wavcoch(
    input_values=waveform_tensor,
    output_hidden_states=True,
    sampling_rate=16000,
).hidden_states[0]

audio = wavcoch.decode_audio(codes)

Notes

This repo includes a bundled vocoder and supports decode_audio(...) for end-to-end waveform synthesis.

mode="causal" is the default and uses the original 921 samples of left padding. mode="centered" redistributes the same total padding as 421 samples on the left and 500 samples on the right. This keeps frame counts unchanged while moving the analysis-window center forward by 500 samples (31.25 ms at 16 kHz). Centered encoding uses future waveform context and is therefore not streaming-causal.

The mode argument is accepted by forward(...), quantize(...), and wav2coch(...). Decoding methods operate on codes and do not take a mode.

Compatibility and validation

This repository uses the exact model.safetensors weights from TuKoResearch/WavCochCausalV8192-vocoder; only the self-contained runtime, configuration metadata, and model card add the encoding-mode API.

Validation used float32 with TF32 disabled:

• Default encoding and explicit mode="causal" produced bit-exact codes, hidden states, decoded cochleagrams, and decoded audio relative to the base model.
• Exact causal compatibility held for input lengths both divisible and not divisible by the 80-sample hop.
• Six synthetic beep configurations produced the same 200 frames for causal and centered modes from one second of audio.
• Centered decoded-cochleagram power advanced by 6 frames (30 ms, the nearest 5 ms token-grid value to 31.25 ms), with shifted-curve Pearson correlations from 0.986 to 0.999.
• AuriStream7BDeep-40Pred float32 head-1 and head-40 surprisals were bit-exact between the base model's codes and this model's causal codes.

Centered mode changes the encoded token sequence and uses approximately 31.25 ms of future waveform context relative to token endpoints. Models trained on causal WavCoch codes should therefore be evaluated explicitly before centered codes are used for likelihood estimates or generation.

An end-to-end impulse diagnostic found that the learned causal decoder can move decoded power peaks about 5 ms later than the geometrically centered front end. The experimental 341-left/580-right compensation is intentionally not exposed as a named mode in this checkpoint; centered means geometric endpoint centering with 421-left/500-right padding.

When called with output_hidden_states=True, WavCoch exposes a single hidden-state layer: the post-FSQ projected embedding sequence used for direct probing.

Loading audio and video files

load_audio() is an optional utility available on both the WavCoch class and loaded instances. It requires the FFmpeg executable on PATH, installed, for example, with conda install -c conda-forge ffmpeg. FFmpeg is only required when this helper is called.

wav = wavcoch.load_audio("random.mp3")  # or .wav, .flac, .m4a, .mp4, .mov, etc.
# wav is a 1D float32 CPU tensor, mono, sampled at 16,000 Hz.
codes = wavcoch.quantize(wav.to(wavcoch.device))

# Optional: decode/downmix/resample without changing the decoded RMS.
wav = wavcoch.load_audio("recording.mp4", normalize_rms=False)

# The same helper works without constructing another model:
wav = type(wavcoch).load_audio("random.mp3")  # WavCoch.load_audio(...) also works.

The helper decodes the first audio stream, downmixes to mono using FFmpeg, and resamples to 16 kHz. It supports the audio/video formats and codecs available in your FFmpeg build; video frames are ignored. Inputs are local file paths, and the complete audio track is loaded into memory. Missing or undecodable audio raises an error.

RMS is measured across the complete waveform after downmixing/resampling. The reference RMS is 0.11 and the allowed range is 0.0011 to 11.0 (0.01x to 100x the reference). Audio within that range is unchanged. Nonzero audio outside it is scaled to the nearest boundary, preserving its relative dynamics; exact silence stays silent. This is not peak normalization: samples are not clipped to [-1, 1]. Use normalize_rms=False to disable this adjustment.

Loading does not tokenize, move the waveform onto a GPU, pad or shift the audio, or change the model's causal/centered mode. Existing tensor-based calls remain unchanged and never invoke this helper automatically. On the centered-capable checkpoint, select centered encoding explicitly with wavcoch.quantize(wav.to(wavcoch.device), mode="centered").

The standalone function can also be imported from wavcoch_audio.py when using the runtime source directly. For a local repository checkout:

from prep_scripts.hf_upload.wavcoch_audio import load_audio

wav = load_audio("random.mp3")

To get this feature from Hugging Face, use the updated revision (or main). Older pinned revisions retain their original API.