Nocturne v1.2 (Teacher) — non-bird bioacoustic species recognition
Nocturne identifies the taxa bird-focused models leave out: insects, amphibians, non-bird mammals and reptiles. v1.2 adds Neotropical anurans and, more importantly, fixes two defects in how v1 was supervised and selected.
v1.2 is the first model in this family to beat the v1 baseline on the frozen benchmark, on both metrics, measured on the identical 13,710-clip held-out test set with identical name-aligned class columns.
| held-out test (calibrated per-class thresholds) | v1 teacher | v1.2 | delta |
|---|---|---|---|
| macro-F1 over all 2,182 core classes | 0.148766 | 0.151051 | +0.002286 |
| mAP (macro) | 0.149849 | 0.154110 | +0.004262 |
| macro-F1 over the 732 classes with test positives | 0.4435 | 0.4503 | +0.0068 |
| top-1 accuracy | 0.4411 | 0.4384 | −0.0027 |
Both improvements exceed the noninferiority margin we pre-committed before scoring (0.00169 macro-F1, 0.00152 mAP), derived from a paired bootstrap over the test rows. They are improvements, not noise.
What actually produced the gain — and what did not
We ran four attempts at v1.2. Three failed to beat v1, and saying which is more useful than the headline:
| attempt | change | core mAP | verdict |
|---|---|---|---|
| v1.2 | + AnuraSet weak labels (recording-level) | 0.1276 | worse — the model learned site signatures, not species |
| v1.2b | + AnuraSet strong labels, medium-quality subset | 0.1486 | no better than v1 |
| v1.2c | + fixed label parser (all call qualities) | 0.1474 | the label fix alone bought nothing |
| v1.2 (this model, internally v1.2d) | fixed parser + multi-hot targets + mAP selection | 0.1541 | passes |
The dataset was never the bottleneck. Two supervision bugs were:
- One positive per training row. The loader emitted one row per (clip, species) pair with a single positive, so a clip containing two species was trained twice, each time teaching the model the other species was absent. Training now uses one multi-hot row per clip. This is the change that produced the gain.
- Checkpoint selection by F1 at a fixed 0.5 threshold. The shipped metric is threshold-calibrated, so selection was optimising the wrong quantity. Selection is now by validation mAP. This model's best epoch was 18 of 40 — the old rule would have shipped a much later, worse checkpoint.
Where it does not improve
On a held-out monitoring site never seen in training (AnuraSet INCT41), v1.2 reaches top-1 0.138. That is roughly ten times better than the weak-label attempt (0.005), but still well below v1's 0.544. Adding AnuraSet has not bought generalisation to a new recording site. If your use case is deploying to an unseen site for anuran monitoring specifically, v1 may still serve you better, and we would rather say so than bury it. Only 3–5 species are scorable at that site, so treat that number as a signal, not a ranking.
Model
| Architecture | Audio Spectrogram Transformer, from MIT/ast-finetuned-audioset-10-10-0.4593 |
| Head | Linear over 2,196 species (multi-label BCE) |
| Input | 10-second mono waveform @ 16 kHz → 128-band log-mel |
| Params | ~86 M |
| Selected | epoch 18 of 40, by validation mAP |
| Trained on | 1× NVIDIA GB10 (DGX Spark), torch 2.13.0, transformers 5.16.1 |
| License | weights CC-BY-4.0, code Apache-2.0 |
Training data
86,703 multi-hot training rows (one per audio file) from InsectSet459, the non-bird portion of iNat Sounds 2024, and AnuraSet strong labels (per-vocalisation time bounds, all call qualities, ≥50 % window overlap). Note the row count is files, not the source datasets' clip totals — an earlier version of this card conflated the two and overstated it.
Evaluation protocol
One frozen harness, soundscape/benchmark.py, scores every checkpoint identically: InsectSet459 + iNat held-out test, one multi-hot row per audio file, per-class thresholds calibrated on validation (grid 0.02→0.94, default 0.3 where a class has no validation positives), macro-averaged over the 2,182 v1 core classes matched by name so models with different vocabularies remain comparable. The harness is validated by reproducing v1's published numbers to four decimals before any comparison is trusted.
macro_f1 averages over all 2,182 columns, but only 732 have positives in the test split, so ~1,450 columns contribute exactly zero regardless of model quality. The companion figure over scorable classes only (0.4503) is the more interpretable one. Both are reported here; neither is cherry-picked.
Files
| file | purpose |
|---|---|
best.pt | selected weights (epoch 18), torch.save state dict |
model.py | standalone strict loader (handles the transformers 5.8→5.16 parameter rename) |
vocab.json | species → class index (2,196) |
thresholds.json | per-class calibrated thresholds |
eval_report.json | full benchmark output, core and anuran slices |
config.yaml, run_manifest.json | exact recipe, seed, code hash, library versions |
Usage
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("stratus-labs/nocturne-v1.2-teacher")
sys.path.insert(0, path)
from model import load_nocturne, predict_file
model, vocab, thresholds = load_nocturne(path) # strict — raises on any key mismatch
for species, score in predict_file(model, "clip.wav", vocab, top_k=5):
print(f"{score:.3f} {species}")
Never load these weights with strict=False. A silent partial load leaves the backbone at AudioSet initialisation and produces plausible-looking nonsense; that bug cost us a full round of invalid benchmarks.
Limitations
- Long-tail. 2,196 classes, 732 with test positives. Rare species are effectively unvalidated.
- Not for birds. Use BirdNET or Perch.
- New-site generalisation for anurans is worse than v1 (see above).
- Not for legal or conservation decisions without field verification.
- Dual-use. Acoustic localisation of rare taxa can aid illegal collection; consider redacting precise GPS in public datasets built with this model.
Related
- Baseline: Nocturne v1 teacher · distilled v1 mini
- Sibling family: Radiance (Sentinel-1 SAR)
- All releases: huggingface.co/stratus-labs
Research page
The write-up and every Nocturne release, in one place: https://runstratus.com/research/nocturne-v1-non-bird-bioacoustics