MOSS Vocal Burst Captioner
An English captioning model for audible voice acting. This self-contained repository includes the complete pinned MOSS-Audio-8B-Instruct base weights, the separately trained LoRA in adapter/, the exact inference implementation, and the executed training recipe and source. The weights are base + adapter, not a silently merged checkpoint.
Start here: the default demo grid
▶ Listen and compare: 50 recordings, MOSS and procedural captions
Our default listening demo shows this model's interval descriptions beside the same original audio, whole-voice descriptions from MOSS Voice Captioner, and the timed script from Procedural Voice Captions. It explains where event timestamps come from, how to seek to them, how the automated review works and which training-text exports are available. The English page opens directly over HTTPS, with embedded audio players.
New: Humanness Voice Annotator — combined GENERAL / SCRIPT demo · Pipeline code and detailed README. This format combines the original MOSS global caption, timed free-form MOSS event descriptions, local Humaneness Ears cues and Parakeet-aligned words. All 50 listening examples include exact endpoints and a new audio-based review, with separate emotion and talking-style criteria. The new automated composite is 3.263/5; its rubric differs from the earlier comparison, so the two composite scores are not directly comparable.
Additional 90-recording MOSS listening room · Companion whole-voice model · Humaneness Ears Medium locator and acoustic scores · Procedural caption code
What this model does
Given the full audio and a start AND end time, this model describes the audible vocal burst in that interval in English. Surrounding audio provides context. It does not itself detect or localize the event.
The acoustic locator is Humaneness Ears Medium, a fine-tuned Whisper Small encoder-only model. It supplies event start/end proposals, 53-way event types, 40 emotion and 57 VoiceNet scores, quality scores, and predicted Orange embeddings. It performs no ASR. Parakeet v3 separately provides transcription and word timestamps when the procedural caption pipeline needs them. The captioner neither consumes a reference caption nor needs a ground-truth event label.
Full annotation pipeline
- Decode the same source audio to mono 16 kHz, retaining the original MP3 and its SHA-256.
- Run Humaneness Ears Medium for numeric scores and candidate burst intervals. Long recordings are covered by multiple windows with timestamps mapped back to the full recording.
- Run the Voice Captioner on the entire audio.
- Run the Vocal Burst Captioner on the entire audio plus each selected interval's start/end seconds. The event is not cropped out of its context. Proposed type labels and script intentions are not input to MOSS.
- Optionally run Parakeet v3 and procedural captions. The latter use measured prediction means/stds and AudioBox Content Enjoyment for listening-pleasantness wording.
- Save all model-generated annotations additively beside the original media. Event proposals and captions are predictions, not human-verified labels.
MOSS requires Transformers 4.57.1; the published Humaneness Ears integration uses Transformers 5.14.1. Use separate environments for these two steps and exchange JSON annotations; mixing the two versions in one interpreter is not supported.
Architecture and training
Base: OpenMOSS-Team/MOSS-Audio-8B-Instruct, pinned revision d4dc5a6d8cd79b43dcd82884c75e303b1ecd016d. MOSS combines an audio encoder at approximately 12.5 audio tokens/s, an audio-to-language adapter, DeepStack features and a Qwen3 language backbone. The frozen base has 9,052,463,488 parameters. Only the language-transformer's attention and MLP LoRA projections are adapted; the audio encoder, audio adapter, DeepStack mergers and original language weights stay frozen.
| Setting | Value |
|---|---|
| Training examples | 80,773 |
| Validation examples | 4,649 |
| Test examples | 4,820 |
| Epochs | 2 |
| Updates | 2,526 |
| LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
| Trainable parameters | 87,293,952 |
| Peak learning rate | 0.0001 |
| Schedule | 5% warmup, cosine decay |
| Optimizer | AdamW, weight decay 0.01 |
| Effective batch | 4 GPUs × 1 example × 16 accumulation = 64 |
| Precision / attention | BF16 / SDPA |
| Loss | Assistant-caption token cross entropy only |
| Audio | Full recording; mono 16 kHz |
| Explicit audio time markers | Enabled every 2 seconds |
| Selected validation loss | 1.423833 |
Supervision comes from machine-generated audio descriptions, not human-adjudicated ground truth. The annotation corpus includes a balanced S1 subset, LAION's Got Talent performances, balanced fine-grained emotion snippets and voice-annotation tasks. Splits are disjoint by exact source-MP3 SHA-256. Event examples from the same recording remain in the same partition. Numeric score vectors, transcripts, intended emotions and reference captions are withheld from the model inputs. The validation loss measures fit to annotation text, not human caption quality or localization accuracy.
Inference: no separate base download required
huggingface-cli download laion/moss-vocal-burst-captioner --local-dir captioner
cd captioner
pip install -r requirements.txt
python inference.py example.mp3 --start 1.2 --end 2.4
For the burst model, both --start and --end are seconds relative to the full recording. A BF16 CUDA GPU is recommended. CPU inference uses float32 and needs substantial memory. Greedy decoding is deterministic, subject to floating-point differences. The training clips were at most 30 seconds; longer full-recording examples in the demo extend beyond that range.
Reproduce training
The repo ships train_adapter.py, the executed trainer and data reader under training/, source hash verification, label masking, the exact task recipe, and vendored official MOSS code under upstream/.
python export_sft.py --help
python -m torch.distributed.run --standalone --nproc_per_node=4 train_adapter.py \
--task burst_caption --data-root prepared --base-model-dir . \
--output training_output --epochs 2
# Resume with --resume training_output/checkpoint-N
Source annotation data: machine-annotated voice corpus. The new synthetic listening/annotation dataset is an inference collection and was not the training set for these adapters. Full optimizer checkpoints remain in the original training run; this release includes the best adapter and training history.
Listening demo and limitations
The demo serves a normal public HTTPS page with inline audio. English and German pairs alternate, beginning with English, and event types are distributed across the page. Whole-recording captions and interval-conditioned captions are shown separately; every displayed interval has start and end seconds. Earlier examples also include untuned MOSS comparisons on the same inputs.
Captions may omit or hallucinate voice details, events or environments. Event proposals require review; class softmax values are not calibrated confidence estimates. Audible delivery does not establish a speaker's identity, actual age, private emotions or medical condition. These two adapters do not replace the separate numeric score and embedding models. The listening page is qualitative inspection, not a human benchmark.
License and credits
Original LoRA weights, project documentation and new annotations: CC BY 4.0, LAION. The redistributed MOSS base weights and official source retain Apache 2.0, with license text in BASE_MODEL_LICENSE and upstream/LICENSE. Attribution and the original source revision are retained. The license of the base is not replaced by the adapter's license.
OpenMOSS source · Humaneness Ears · LoRA · PEFT.
Measured inference recommendations
Use greedy decoding (temperature 0.0). Start the separate Humaneness Ears locator at 0.6, with overlap suppression at half the shorter interval and minimum duration 0.08 s. Use 0.5 for recall or 0.7 for more conservative proposals. This is a provisional practical recommendation; threshold rankings changed between the selection and confirmation clips.
| Ten-clip confirmation setting | Whole-caption fidelity | Event-caption fidelity† | Timing | Precision‡ | Recall‡ | Composite |
|---|---|---|---|---|---|---|
| Threshold 0.5, temperature 0.0 | 3.90 | 3.38 | 3.30 | 3.20 | 3.80 | 3.55 |
| Threshold 0.6, temperature 0.0 | 3.90 | 3.52 | 3.30 | 3.40 | 3.60 | 3.60 |
| Threshold 0.7, temperature 0.0 | 3.90 | 3.54 | 3.20 | 3.70 | 3.10 | 3.41 |
| Threshold 0.5, temperature 0.3 | 3.80 | 3.31 | 3.30 | 3.20 | 3.80 | 3.51 |
| Threshold 0.5, temperature 0.7 | 3.80 | 3.27 | 3.30 | 3.20 | 3.80 | 3.50 |
All values are 0–5 automated audio-reviewer scores, not human benchmark scores. † Conditional event-caption mean; empty event sets are handled separately in the composite. ‡ Reviewer ratings, not numerical precision/recall percentages. The test used 20 complete recordings (10 synthetic acting, 10 Emilia), with a 10/10 selection/confirmation split. The selected 0.5 greedy setting is reported separately from the 0.6 control that topped confirmation. Temperatures 0.3 and 0.7 used top-p 0.9, top-k 0 and one recorded seed; they did not provide consistent gains.
The separate original-50 listening review scored 3.94 whole-caption / 3.68 burst-caption / 3.71 location, with 141 intervals. That is a different sample set and protocol, not a paired improvement claim.
Complete recommendations, grid, scoring rubric and limitations · Machine-readable settings · Detailed decoding results · Separate procedural-caption tuning.
Timed training surfaces and same-audio comparison
▶ 50 audios: MOSS versus procedural captions, with six selectable CAPTION/TRANSCRIPT variants
Both captioners were rerun on the same 50 complete recordings using greedy decoding, locator threshold 0.6, 0.08-s minimum duration, suppression at half the shorter interval and no selected-event cap. Intended script, scenario and emotion labels were withheld. A blinded automated audio review used the same five-part rubric for both pipelines.
| Pipeline | Composite / 5 | Global fidelity | Burst description | Timing | Event precision rating | Event recall rating |
|---|---|---|---|---|---|---|
| MOSS, greedy and uncapped | 3.863 | 4.34 | 3.72 | 3.46 | 3.50 | 4.14 |
| Procedural, compact tags | 2.704 | 2.74 | 2.02 | 3.04 | 3.06 | 3.10 |
These are Gemini audio-judge ratings, not human scores or precision/recall percentages. The 25-clip confirmation composites are 3.646 for MOSS and 2.760 for procedural captions. Overall procedural compact tags tied the previous prose. The earlier original-50 scores and twenty-clip tuning use different protocols; do not compare their numbers as a measured improvement. Frozen metrics and rubric.
training_formats.py combines the unchanged whole-audio caption, optionally followed by interval-conditioned burst descriptions, with plain ASR, inline burst ASR or fully timed ASR. It creates six CAPTION/TRANSCRIPT choices. Every burst has both endpoints and a duration; overlapping durations are included within the enclosing segment budget, never counted twice. ASR is supplied separately by Parakeet v3. Exact Humaneness Voice format · Humaneness Voice Small · Procedural source.
Fully annotated synthetic acting release
German-English Voice Acting Challenges contains 38,427 recordings / 314.13 hours, in 151 verified WebDataset TARs, under CC BY 4.0. Every sample has both MOSS pipelines, Humaneness Ears Medium scores and embeddings, Parakeet ASR, nine procedural wording variants, six selectable CAPTION/TRANSCRIPT training formats, and the complete original generation voice/prompt/model metadata without credentials. These recordings are an inference release, not this adapter's training corpus.
▶ One complete audio annotation, with every target and training choice · Paired 50-audio evaluation · Reproduction instructions.