MATPAC++ (AudioSet fine-tuned encoder)
MATPAC (Masked latent Prediction And Classification) is a self-supervised audio
and music encoder. This repo hosts the MATPAC++ encoder, self-supervised then
fine-tuned on AudioSet, encoder only (matpac_plus_as_48_1_map_enconly.pt), a
transformers-native, trust_remote_code port of the official inference code
(aurianworld/matpac, folder
inference_matpac).
- 📝 MATPAC: Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
- 📝 MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning
If you want ready-made AudioSet class predictions instead of embeddings, use matpac_audioset_finetune_classifier (same fine-tuning, plus the 527-way classification head).
The MATPAC family
| Repo | Trained on | Output | Notes |
|---|---|---|---|
| matpac_general_audio | General audio (AudioSet) | 3840-d embeddings | Default general-purpose audio encoder |
| matpac_music | Music | 3840-d embeddings | Music-specialized encoder |
| matpac_audioset_finetune_encoder (this repo) | AudioSet (fine-tuned) | 3840-d embeddings | Fine-tuned encoder, no classification head |
| matpac_audioset_finetune_classifier | AudioSet (fine-tuned) | 527-way logits | Ready-to-use AudioSet tagger |
Usage
from transformers import AutoModel, Wav2Vec2FeatureExtractor
import torch
import torchaudio
model = AutoModel.from_pretrained("auriankelen/matpac_audioset_finetune_encoder", trust_remote_code=True)
processor = Wav2Vec2FeatureExtractor.from_pretrained("auriankelen/matpac_audioset_finetune_encoder", trust_remote_code=True)
waveform, sr = torchaudio.load("my_file.wav")
waveform = waveform.mean(dim=0) # mono
resample_rate = processor.sampling_rate
if resample_rate != sr:
waveform = torchaudio.transforms.Resample(sr, resample_rate)(waveform)
inputs = processor(waveform, sampling_rate=resample_rate, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# outputs.last_hidden_state: (batch, 3840) -- mean-pooled over time by default
# outputs.hidden_states: tuple of 12 tensors, one per encoder layer, each (batch, 3840)
all_layer_hidden_states = torch.stack(outputs.hidden_states)
print(all_layer_hidden_states.shape) # [12, batch, 3840]
Keeping the time dimension / faster inference
# Keep the time axis instead of mean-pooling it
outputs = model(**inputs, pull_time_dimension=False)
# outputs.last_hidden_state: (batch, time, 3840)
# Vectorized inference: faster on large batches, adds some padding so it is
# slightly less precise than the default ("precise", used for the paper's results)
outputs = model(**inputs, inference_type="fast")
Requirements
This model needs a couple of packages beyond transformers to run its
trust_remote_code files: torchaudio, einops, and timm==0.4.12 (the exact
timm version the checkpoints were exported with the internal naming of the ViT
block submodules must match for the weights to load correctly).
Citation
@inproceedings{quelennec2025matpac,
title={Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning},
author={Quelennec, Aurian and Chouteau, Pierre and Peeters, Geoffroy and Essid, Slim},
booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
year={2025},
url={https://ieeexplore.ieee.org/document/10887666},
doi={10.1109/ICASSP49660.2025.10887666}
}
@article{quelennec2025matpacenhancedmaskedlatent,
title={MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning},
author={Aurian Quelennec and Pierre Chouteau and Geoffroy Peeters and Slim Essid},
journal={arXiv preprint arXiv:2508.12709},
year={2025},
url={https://arxiv.org/abs/2508.12709}
}
Credits
- aurianworld/matpac for the original training and inference code.
- Fairseq for the training framework.
- M2D for the base of the inference code.
- DINO for the classification head.