MERT-v1-95M Music Emotion Recognition (ONNX INT8)

Model Architecture: m-a-p/MERT-v1-95M (Music Foundation Transformer, ICLR 2024) + Temporal Attention Pooling → Dynamic Quantized INT8 ONNX
Target Task: Continuous 2D Valence & Arousal (VA) Prediction in Russell's Circumplex Model [0.0, 1.0]
Optimized For: Ultra-fast CPU Inference, Light Memory Footprint (~111 MB RAM), Serverless / Microservice Deployment

1. Model Overview

This model is a specialized Music Emotion Recognition (MER) regression engine developed for affective computing and mental health music therapy recommendation systems (part of the SE121-microservices project).

It predicts continuous 2D coordinates:

• Valence (V ∈ [0.0, 1.0]): Measures musical pleasantness / emotional positivity (0.0 = deeply sad/melancholic, 0.5 = neutral, 1.0 = highly positive/happy).
• Arousal (A ∈ [0.0, 1.0]): Measures energetic intensity / stimulation (0.0 = sleepy/calm, 0.5 = moderate, 1.0 = intense/excited).

Russell's Circumplex Emotion Mapping

Neutral Center: (0.5, 0.5)

2. Experimental Benchmark Results

Evaluated independently on a Held-Out Test Set of 386 songs (isolated 15% split from DEAM and PMEmo datasets, total N = 2,569 tracks):

======================================================================
           EVALUATION ON HELD-OUT TEST SET (386 TRACKS)
======================================================================
Valence CCC (Lin's Correlation) : 0.7358  (Approaching Human Agreement Ceiling)
Valence R² Score                : 0.4783
Valence MAE (Mean Absolute Err) : 0.0895
----------------------------------------------------------------------
Arousal CCC (Lin's Correlation) : 0.8204  (SOTA Performance)
Arousal R² Score                : 0.6575
Arousal MAE (Mean Absolute Err) : 0.0861
======================================================================

3. Audio Preprocessing Protocol

• Input Slicing: 15-second representative segment extracted from the song's center ([T/2 - 7.5s, T/2 + 7.5s]), leveraging the Thin-Slicing Paradigm and Chorus Salience Principle.
• Sampling Rate: 24,000 Hz Mono audio.
• Input Tensor Shape: [1, 360000] float32 tensor (24,000 × 15 = 360,000 samples).
• ONNX Input Name: audio_waveform
• ONNX Output Name: valence_arousal (Shape: [1, 2])

4. Quick Start & Inference Example

Installation

pip install onnxruntime soundfile scipy numpy huggingface_hub

Python Inference Code

import onnxruntime as ort
import soundfile as sf
import scipy.signal
import numpy as np
from huggingface_hub import hf_hub_download

# 1. Download model weights from Hugging Face Hub
model_path = hf_hub_download(
    repo_id="huyleit/mert-v1-95m-music-emotion-int8",
    filename="mert_emotion_int8.onnx"
)

# 2. Initialize ONNX Runtime Session
opts = ort.SessionOptions()
opts.intra_op_num_threads = 4
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
session = ort.InferenceSession(model_path, sess_options=opts, providers=['CPUExecutionProvider'])

def predict_music_emotion(audio_path: str) -> dict:
    target_sr = 24000
    duration_sec = 15
    target_samples = target_sr * duration_sec

    # Read audio
    data, sr = sf.read(audio_path, dtype='float32')
    if data.ndim > 1:
        data = np.mean(data, axis=1) # Mono channel

    # Extract 15s center segment
    orig_samples = int(sr * duration_sec)
    if len(data) > orig_samples:
        start = (len(data) - orig_samples) // 2
        data = data[start:start + orig_samples]
    elif len(data) < orig_samples:
        data = np.pad(data, (0, orig_samples - len(data)))

    # Resample to 24kHz
    if sr != target_sr:
        data = scipy.signal.resample(data, target_samples).astype(np.float32)

    input_tensor = np.expand_dims(data, axis=0).astype(np.float32)

    # Inference
    outputs = session.run(["valence_arousal"], {"audio_waveform": input_tensor})
    valence, arousal = float(outputs[0][0][0]), float(outputs[0][0][1])

    # Classify Russell Quadrant
    quadrant = "Q1" if valence >= 0.5 and arousal >= 0.5 else \
               "Q2" if valence < 0.5 and arousal >= 0.5 else \
               "Q3" if valence < 0.5 and arousal < 0.5 else "Q4"

    return {
        "valence": round(valence, 4),
        "arousal": round(arousal, 4),
        "quadrant": quadrant
    }

# Example usage:
# result = predict_music_emotion("sample_song.mp3")
# print(result)
# Output: {'valence': 0.6011, 'arousal': 0.5947, 'quadrant': 'Q1'}

5. Training Details

• Base Model: m-a-p/MERT-v1-95M
• Pretraining Data: 160,000 hours of acoustic music
• Fine-tuning Datasets: DEAM (1,802 tracks) + PMEmo (767 tracks) = 2,569 total tracks
• Loss Function: Combined Lin's Concordance Correlation Coefficient (CCC) + MSE:

  Loss_total = 0.75 * Loss_CCC + 0.25 * Loss_MSE
  

• Optimization: AdamW (lr = 1e-4), Cosine Annealing, Batch Size = 4, Mixed Precision (AMP FP16)
• Quantization: Dynamic INT8 Quantization via ONNX Runtime

6. Citation & References

@inproceedings{li2024mert,
  title={MERT: Acoustic Music Understanding with Large-Scale Self-supervised Training},
  author={Li, Yizhi and Yuan, Ruibin and Zhang, Ge and Ma, Yinghao and Lin, Chenghua and Chen, Xingran and Ragni, Anton and Benetos, Emmanouil and Gyenge, Norbert and Dannenberg, Roger and others},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2024}
}

@article{russell1980circumplex,
  title={A circumplex model of affect.},
  author={Russell, James A},
  journal={Journal of personality and social psychology},
  volume={39},
  number={6},
  pages={1161},
  year={1980}
}