MiDashengLM-0.6B fine-tuned on ESC-50

mispeech/midashenglm-0.6b-fp32 adapted with LoRA to produce a one-sentence description of the main environmental sound in a clip (ESC-50, 50 classes). Weights: merged (LoRA folded into the base weights, fp32).

Model

Training data

• Dataset: ESC-50 via TigreGotico/ESC-50 (2000 clips x 5 s, 44.1 kHz, 50 classes, 40 clips/class; labels and folds from meta/esc50.csv).
• Official folds: train = folds 1-3 (1200 clips), validation = fold 4 (400, checkpoint selection), test = fold 5 (400).
• Audio is resampled 44.1 kHz -> 16 kHz (torchaudio.functional.resample).
• Targets are generated from the class labels, not human-written captions: 3 fixed paraphrases per class (e.g. dog -> "A dog is barking."), sampled at random per step; they state only what the label says.
• Instructions (one sampled per step):
  • "Describe the main environmental sound in this audio. Give a short description without inventing details."
  • "What environmental sound is present in this recording?"
  • "Identify the main sound event in the audio and describe it briefly."
  • "Listen to the audio and give a short description of the main sound."

Prompt format (model chat template; <|AUDIO|> is expanded to one token per audio frame):

<|im_start|>user
{instruction}<|audio_bos|><|AUDIO|><|audio_eos|><|im_end|>
<|im_start|>assistant
{target}<|im_end|>

Loss is computed on the answer tokens and the closing <|im_end|> only.

Training procedure

Evaluation (ESC-50 fold 5, 400 clips)

Same four instructions for both models (INSTRUCTIONS[clip_index % 4]), greedy decoding. The generated sentence is mapped to a class (exact paraphrase match, otherwise the earliest class keyword); unmappable outputs count as errors. ROUGE-L / BLEU use all paraphrases of the true class as references and are secondary because the references are templated.

Usage

Requires trust_remote_code=True. With transformers >= 5 the remote code's non-persistent STFT window and mel-filterbank buffers are left uninitialised by meta-device loading (output degenerates to !!!!); repair_frontend_buffers below recomputes them. It is a no-op fix on transformers 4.x.

import torch, torchaudio, torchaudio.functional as AF
from transformers import AutoModelForCausalLM, AutoProcessor


def repair_frontend_buffers(model):
    for m in model.modules():
        name = type(m).__name__
        if name == "DashengFrontend":
            c = m.config
            m.spectrogram_window = torch.hann_window(c.win_length)
            m.melscale_fbanks = AF.melscale_fbanks(c.n_fft // 2 + 1, c.f_min, c.f_max, c.n_mels, c.sample_rate)
        elif name == "AcousticFrontend":
            m.spectrogram_window = torch.hann_window(m.win_length)
            m.melscale_fbanks = AF.melscale_fbanks(m.n_fft // 2 + 1, 0.0, m.sample_rate / 2, m.n_mels, m.sample_rate)


model = AutoModelForCausalLM.from_pretrained(
    "thealper2/midashenglm-0.6b-esc50", trust_remote_code=True, dtype=torch.float32
)
repair_frontend_buffers(model)
model = model.cuda().eval()
processor = AutoProcessor.from_pretrained("mispeech/midashenglm-0.6b-fp32", trust_remote_code=True)

wav, sr = torchaudio.load("clip.wav")
wav = AF.resample(wav.mean(0), sr, 16000)
messages = [{"role": "user", "content": [
    {"type": "text", "text": "Describe the main environmental sound in this audio. Give a short description without inventing details."},
    {"type": "audio", "audio": wav.numpy()},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], audio=[wav.numpy()], return_tensors="pt").to("cuda")
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):  # weights must stay fp32
    out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(processor.tokenizer.decode(out[0], skip_special_tokens=True))

Do not cast the model to bf16/fp16: the acoustic front-end returns float32 and a half-precision conv then fails. Use fp32 weights with autocast as above.

Limitations

• Trained and evaluated only on ESC-50: single-label, 5-second clips from Freesound with one dominant event.
• Output vocabulary is effectively limited to the 50 ESC-50 classes and 3 phrasings each; it does not describe scenes, sources, counts or acoustic properties.
• Accuracy/macro-F1 depend on the text-to-class mapping; ROUGE-L/BLEU mostly measure agreement with the templates.
• Results are from a single run with one seed on one fold split (not 5-fold cross-validation).

Training data: ESC-50 by Karol J. Piczak (CC BY-NC 3.0). Base model: MiDashengLM-0.6B by Xiaomi (Apache-2.0).