MATPAC++ (AudioSet classifier)

MATPAC (Masked latent Prediction And Classification) is a self-supervised audio and music encoder. This repo hosts the MATPAC++ encoder, fine-tuned on AudioSet, with its 527-way classification head (matpac_plus_as_48_1_map_enc_and_head.pt), a transformers-native, trust_remote_code port of the official inference code (aurianworld/matpac, folder inference_matpac).

• 📝 MATPAC: Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
• 📝 MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning

If you want raw embeddings instead of AudioSet class predictions, use matpac_audioset_finetune_encoder (same fine-tuning, no classification head).

The MATPAC family

Usage

from transformers import AutoModel, Wav2Vec2FeatureExtractor
import torch
import torchaudio

model = AutoModel.from_pretrained("auriankelen/matpac_audioset_finetune_classifier", trust_remote_code=True)
processor = Wav2Vec2FeatureExtractor.from_pretrained("auriankelen/matpac_audioset_finetune_classifier", trust_remote_code=True)

waveform, sr = torchaudio.load("my_file.wav")
waveform = waveform.mean(dim=0)  # mono

resample_rate = processor.sampling_rate
if resample_rate != sr:
    waveform = torchaudio.transforms.Resample(sr, resample_rate)(waveform)

inputs = processor(waveform, sampling_rate=resample_rate, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

probs = outputs.logits.softmax(dim=-1)
values, indices = probs.topk(k=5, dim=-1)
labels = [
    f"{model.config.id2label[i.item()]} ({v:.2f})"
    for i, v in zip(indices[0], values[0])
]
print(labels)
# -> ['Heart sounds, heartbeat (0.62)', 'Speech (0.18)', 'Heart murmur (0.14)', ...]

Embeddings (outputs.last_hidden_state, outputs.hidden_states) are still available from the same forward call, exactly as in the other MATPAC repos.

Demo: what fires, and when

matpac_youtube_app.py runs the classifier on any YouTube video and shows which AudioSet classes activate over time, with the heatmap playhead following the video so you can hear what each class is reacting to.

pip install torch torchaudio soundfile transformers einops timm==0.4.12 flask yt-dlp
python matpac_youtube_app.py   # then open the URL it prints

It picks a random video for you, or takes a pasted URL. Running it from a clone of this repo uses the model files sitting next to it, so it needs no download and no token; --repo auriankelen/matpac_audioset_finetune_classifier pulls from the Hub instead. Needs ffmpeg on PATH; only the first 120s of audio are analysed.

Requirements

This model needs a couple of packages beyond transformers to run its trust_remote_code files: torchaudio, einops, and timm==0.4.12 (the exact timm version the checkpoints were exported with the internal naming of the ViT block submodules must match for the weights to load correctly).

Note that torchaudio.load (used in the examples above) needs a decoding backend: install soundfile, or torchcodec on torchaudio >= 2.9, or reading the .wav will fail with Couldn't find appropriate backend to handle uri ....

Citation

@inproceedings{quelennec2025matpac,
  title={Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning},
  author={Quelennec, Aurian and Chouteau, Pierre and Peeters, Geoffroy and Essid, Slim},
  booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2025},
  url={https://ieeexplore.ieee.org/document/10887666},
  doi={10.1109/ICASSP49660.2025.10887666}
}

@article{quelennec2025matpacenhancedmaskedlatent,
  title={MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning},
  author={Aurian Quelennec and Pierre Chouteau and Geoffroy Peeters and Slim Essid},
  journal={arXiv preprint arXiv:2508.12709},
  year={2025},
  url={https://arxiv.org/abs/2508.12709}
}

Credits

• aurianworld/matpac for the original training and inference code.
• Fairseq for the training framework.
• M2D for the base of the inference code.
• DINO for the classification head.