Model card for naflexclap_base_pf8_pt16_moderntextp.160m

Four NaFlexClap checkpoints benchmarked on the standalone CLAP suite + MAEB(beta)-30.

Methodology

• Standalone CLAP suite: audio uncapped (batch-max padding); classification rows show default prompt ("This is a sound of {c}.") and best-of-sweep; retrieval is A2T/T2A = text-retrieval / audio-retrieval R@5.
• MAEB(beta)-30: --batch_size 16, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, uncapped except BirdCLEF capped at 60 s for comparability (A 0.604 was pre-capped; B/C/D recomputed at 60 s).
• Precision: fp32 weights+activations, no autocast, TF32 matmuls (the MAEB encoder default). Not a precision artifact.

1. Architecture, params, geometry

Audio tower = embed/depth/heads; text tower = width/layers/heads.

What each comparison isolates:

• A vs D — text tower only (bidir+MAP+qk-norm vs causal+EOS+bias), everything else equal. Cleanest.
• A vs C — audio trunk (768/12 ungated GELU vs 512/20 gated GELU) + geometry (pf8/pt16 vs pf4/pt20).
• A vs B — confounded 3 ways: pt16→pt20, 160M→240M samples, audio GELU→SwiGLU. Not attributable to one.

2. Standalone CLAP suite

A2T/T2A = text-retrieval / audio-retrieval R@5.

Classification is prompt-sensitive: default "This is a sound of {c}." is pathological for some checkpoints (D UrbanSound 0.495→0.709 best; C 0.456→0.775 best). Best-prompt is the fairer signal. GTZAN best uses "This is a {c} song."/"a {c} song"; UrbanSound/ESC-50 best vary per model.

‡ UrbanSound rows use the full danavery/urbansound8K (8732 clips), matched data for zero-shot + probe.

Probe = official UrbanSound8K protocol: supervised logreg on audio embeddings, 10-fold CV on the predefined fold splits (train 9 / test 1, ×10, averaged) — literature-comparable. All four land at 0.838–0.856 (1.8 pt spread) despite zero-shot ranging 0.46–0.80, so the entire UrbanSound zero-shot spread is text-alignment / prompt, not audio quality: e.g. D probes 0.8441 (above B and C) yet its default zero-shot was 0.494 (EOS-pooling text misalignment; audio is fine). (A naive random 5-fold inflates the probe to ~0.93 via same-recording leakage — do not use it; the UrbanSound8K authors explicitly warn against reshuffling.)

¶ VGGSound = SINGLE-label, metric is top-1/top-5 accuracy (309 classes; not mAP). Source mteb/VGGSound test (9,888 clips, audio + integer label, ready 309-class ClassLabel vocab — no reconstruction); audio uncapped; VGGSound prefers a template (bare {} worst 0.22–0.26); B/C/D peak at "This is a sound of {c}.", A at "the sound of {c}"; ordering B > C > D > A (A's text tower aligns differently, ~6 pt best-to-best). Harness validated against LAION-CLAP (laion/clap-htsat-unfused through the same pipeline = 0.234 top-1, ≈ its published VGGSound zero-shot ~0.26).

3. MAEB(beta)-30 — full per-task (main_score)

† UrbanSound8KT2ARetrieval ≈ 0.009 is degenerate task data (10 labels, 1-to-1) — not a model signal.

4. Audio-representation probes (supervised, prompt-independent)

Two supervised linear probes (logreg) on the audio embeddings — isolating the audio tower from text/prompt alignment, to test whether the wide MAEB / zero-shot spreads reflect real audio-representation differences.

• VoxPopuli gender — facebook/voxpopuli test, all 16 language configs (n = 16,991, 69% male, 1133 speakers). Speaker-disjoint GroupKFold(5) (same speaker ⇒ same gender, else a random split leaks), class_weight=balanced; read straight from the cached parquet (no datasets-arrow rebuild).
• UrbanSound8K — the official 10-fold probe from §2 (full danavery 8732, predefined folds), shown here for the audio-representation view.

All four audio towers encode gender identically — AUC 0.987–0.988 (0.10 pt spread), bacc 0.967–0.970. The three metrics agree and rank the models as a tie (imbalance doesn't pull them apart because separability is so high; AUC = cleanest "is-it-separable" number, raw accuracy would be the trap). This is the second probe (after UrbanSound, 0.838–0.856, 1.8 pt spread) showing the audio towers are equivalent — so the large MAEB spreads on VoxPopuliGenderClustering (0.13–0.39) and UrbanSound8KT2ARetrieval were task/metric artifacts (500-sample clustering noise; degenerate 1-to-1 retrieval), not audio-representation gaps. Note D (causal/EOS) is marginally best on both probes despite its weak MAEB clustering — its weaknesses are entirely text-side. (MAEB's gender task uses mteb/mini-voxpopuli, only 500 samples across 5 langs — hence the noise.)

Key findings

1. Text tower (A vs D, clean isolation): bidirectional+MAP > causal+EOS — MAEB 0.4545 vs 0.4276 (−2.7 pt). The loss concentrates on rich-text retrieval (CommonVoice −0.20, Fleurs −0.20, JamAltLyric −0.15, Clotho, SpokenSQuAD, MInDS14): EOS pooling compresses the whole caption into the last token; MAP keeps the full sequence retrieval needs. Even/better on short-label classification (SpeechCommands +0.04, CREMA_D +0.04). Causal+EOS is also more prompt-fragile (its weak default-prompt zero-shot undersells it).

2. Longer training (A vs B, +confounds): helps the CLAP suite, flat on MAEB. B is best on ESC-50/AudioCaps/ Clotho/BirdCLEF but ties A on MAEB (0.4533 vs 0.4545) — two big drops (SpeechCommands −0.17, VoxPopuliGenderClustering −0.19) cancel broad gains; excluding those two, B is +1.2 pt over A on the other 28. B's SpeechCommands −0.17 is checkpoint-specific (D, also causal-free pt16, is +0.04 there).

3. mediumd audio trunk (A vs C): −2.1 pt MAEB, but C is 17% smaller (139.7M) and the gap is task-dependent (wins BirdCLEF/GTZAN, loses speech/temporally-fine). No robust pt16-vs-pt20 effect once prompts are matched.

4. No pt20 UrbanSound weakness and no fill/precision artifacts — UrbanSound clips are ≤4 s (~21 tokens), too short for pt to matter; per-class pt20 gains on jackhammer; pad-mode (floor/silence/repeat) moves SpeechCommands <0.003; MAEB runs fp32.

Result provenance (persistent)

• Eval scripts (under those dirs): clapv2_cls_sweep.py, gtzan_sweep.py, birdclef_capped.py (argv model/ckpt/cap/out), urban_perclass.py, urban_durations.py, speechcmd_padmode.py, urban_full.py (full-8732 danavery: zero-shot sweep + official 10-fold probe in one embedding pass; urban_probe10.py = probe-only; urban_probe.py = old random-5-fold, superseded); vox_gender_probe.py (VoxPopuli gender, speaker-disjoint GroupKFold; VOX_LANGS env selects configs; results in voxgender.log = 5-lang, voxgender16.log = full 16-lang).

Appendix — pf8 vs pf4 (frequency tokens): confounded, but a directional signal

Can we isolate a pf effect? Not cleanly. pf4 appears in exactly one checkpoint — C (mediumd_pf4_pt20) — which differs from the base pf8 models by more than pf: a narrower trunk (mediumd 512 vs base 768) and gated audio attention (only C). pt is not a confounder here — B (base_pf8_pt20) vs A (base_pf8_pt16) isolates pt and came out ~neutral — so C-vs-base = pf4 ⊕ trunk ⊕ gating, with trunk + gating the live confounds.

But the pattern isn't random. Against A (the clean pair — pf differs, same 160M-sample training):

Caveats (why "suggestive", not proven): gating is co-located in C and could drive (some of) the spectral wins; and the win costs 2× freq tokens — a resolution/compute trade, not free.

To settle it: one controlled run — base-pf4-pt20 (768 trunk, gating off, same recipe) vs B (base-pf8-pt20) — neutralizes trunk + gating + pt and leaves pf as the only mover. Expected from the above: small gains on spectral / tonal tasks, ~flat-to-slightly-down elsewhere, at 2× the freq-token cost.