ചെവി Mozhi Chevi

Chevi (ചെവി) means ear. This is a single 1.94 MB ONNX model that listens to Malayalam speech and tells you how it is spoken:

• style tags: gender, pitch, speaking rate, expressiveness, recording quality, emotion
• a caption built from those tags, in the same style as the dataset's prompt-TTS captions
• measurements: Median F0, F0 variability, Syllables / s, PESQ, STOI, SI-SDR, where PESQ / STOI / SI-SDR are estimated without a clean reference
• a 128-d style embedding: clips with similar delivery land close together

It takes raw 16 kHz audio (the spectrogram is part of the graph), runs in about 5 ms for 3 s of audio on one CPU thread, and was trained on a CPU in 26 minutes.

Take Demo here - https://huggingface.co/spaces/nuhmanpk/mozhi-chevi-demo

What it hears

Results

Test split of nuhmanpk/mozhi-malayalam-speech, 767 clips. Two baselines keep this honest: overall majority always answers the most common label, and per-voice majority knows who is speaking and answers that speaker's usual label. Beating the per-voice baseline means the model hears the clip, not just the voice.

Voices it has never heard

A second copy was trained with 3 voices removed (mlf_04310, mlm_03993, WnrwjCDN1PgNuCK4IrstUpDLJB93) and tested on 172 of their clips. This is the number that matters for new speakers or TTS output.

Use it

Everything needed to read the outputs (label lists, units, caption template) is stored inside the .onnx file as metadata under the key mozhi_chevi.

Python

import json, numpy as np, onnxruntime as ort, soundfile as sf, librosa
from huggingface_hub import hf_hub_download

sess = ort.InferenceSession(hf_hub_download("nuhmanpk/mozhi-chevi", "mozhi_chevi.onnx"))
spec = json.loads(sess.get_modelmeta().custom_metadata_map["mozhi_chevi"])

audio, sr = sf.read("clip.wav", dtype="float32", always_2d=True)
audio = librosa.resample(audio.mean(1), orig_sr=sr, target_sr=16000)
out = dict(zip([o.name for o in sess.get_outputs()], sess.run(None, {"audio": audio[None]})))

tags = {k: v[int(out[k][0].argmax())] for k, v in spec["tags"].items()}
measures = {k: float(out[k][0]) for k in spec["measures"]}
print(tags)
print(measures)   # f0_median Hz, syl_per_sec, pesq, stoi, si_sdr dB ...

Browser (onnxruntime-web), live from a microphone or a file

import * as ort from "onnxruntime-web";
const s = await ort.InferenceSession.create("https://huggingface.co/nuhmanpk/mozhi-chevi/resolve/main/mozhi_chevi.onnx");
// samples: Float32Array, mono 16 kHz (decode + resample with an OfflineAudioContext)
const out = await s.run({ audio: new ort.Tensor("float32", samples, [1, samples.length]) });
const argmax = a => a.indexOf(Math.max(...a));
const TAGS = {"gender": ["male", "female", "unknown"], "pitch": ["very low-pitch", "low-pitch", "slightly low-pitch", "slightly high-pitch", "moderate pitch", "high-pitch", "very high-pitch"], "speaking_rate": ["very slowly", "slowly", "slightly slowly", "at a moderate speed", "slightly fast", "fast", "very fast"], "expressiveness": ["very monotone", "monotone", "slightly expressive and animated", "expressive and animated", "very expressive and animated"], "recording_quality": ["very noisy", "noisy", "slightly noisy", "balanced in clarity", "slightly clean", "clean", "very clean"], "emotion_label": ["neutral", "happy", "sad", "disgusted", "surprised", "angry"]};
const tags = Object.fromEntries(Object.entries(TAGS).map(([k, v]) => [k, v[argmax([...out[k].data])]]));
const pesq = out.pesq?.data[0], rate = out.syl_per_sec?.data[0];

Ideas

• Judge a TTS model without references. Run Chevi on synthesised speech: estimated PESQ/STOI/SI-SDR, pace and pitch tell you whether a checkpoint got better, and whether the speed control really changes syllables per second.
• Caption new recordings for prompt-based TTS training, in the same vocabulary as this dataset.
• Filter data: drop clips Chevi rates as noisy or low-PESQ before training anything.
• Live feedback in a browser page: a reading-practice meter that shows pace, pitch range and recording quality as someone speaks.
• Search by delivery: nearest neighbours in the style embedding find clips spoken the same way.

How it was made

• Data: 17444 train / 754 validation / 767 test clips (26.2 h, 124 speakers, sources imasc, openslr63, smc_msc), using the dataset's own splits. Training clips are capped per speaker.
• Targets: the dataset's per-clip annotations. Tags rarer than 30 training clips are ignored; ordered tags (pitch, rate, expressiveness, quality) are sorted by their underlying measurement.
• Model: fixed-DFT log-mel front end (80 bands) → dilated TDNN with residual blocks → mean+std pooling → 128-d style vector → one softmax per tag + one regression layer. 0.30 M parameters.
• Training: CPU only (Intel(R) Xeon(R) CPU @ 2.20GHz, 2 threads), 20 epochs × 150 steps, batch 64, random 2 to 5 s crops, time masking, AdamW + one-cycle LR (0.003), class-balanced cross-entropy + Smooth-L1. Best epoch by validation: 20.
• ONNX check: outputs match PyTorch to a relative difference of 3.43e-06.

Limits

• Chevi imitates the dataset's automatic annotations. Where those are noisy (emotion especially, which is mostly neutral), so is Chevi.
• Quality numbers are estimates learned from studio read speech; treat them as relative scores, not lab-grade PESQ or STOI.
• Tags such as pitch partly follow the speaker; the unseen-voice table above shows how far it carries to new voices.
• Built for Malayalam read speech at 16 kHz. Other languages, singing, or heavy music will be outside what it has heard.

License and citation

Trained on nuhmanpk/mozhi-malayalam-speech (licenses in the training data: CC-BY-SA-4.0); this model follows the same terms. The IMaSC recordings come from:

@article{gopinath2022imasc,
  title   = {IMaSC -- ICFOSS Malayalam Speech Corpus},
  author  = {Gopinath, Deepa P and D K, Thennal and Nair, Vrinda V and K S, Swaraj and G, Sachin},
  journal = {arXiv preprint arXiv:2211.12796},
  year    = {2022}
}

Card generated 2026-10-07.