Liujgoj-Cantonese-Qwen2.5-Omni-7B-ASR-V2
A Cantonese Automatic Speech Recognition model that directly transcribes spoken Cantonese into Liujgoj Cantonese Orthography (溜歌粵語羅馬字).
Overview
Liujgoj-Cantonese-Qwen2.5-Omni-7B-ASR-V2 is a specialized Automatic Speech Recognition (ASR) model built through a three-stage training pipeline:
Qwen2.5-Omni-7B
│
▼
Liujgoj Cantonese CPT
│
▼
Liujgoj Cantonese SFT
│
▼
Liujgoj Cantonese ASR
Starting from Qwen2.5-Omni-7B, the model was first adapted to the Liujgoj Cantonese language space through Continued Pretraining (CPT), then further structured through Supervised Fine-Tuning (SFT), and finally fine-tuned for Automatic Speech Recognition.
The final model is designed to perform:
Spoken Cantonese Audio → Liujgoj Cantonese
Unlike conventional Cantonese ASR systems that typically transcribe speech into Chinese characters, this model directly generates Liujgoj Romanized Cantonese, including syllable structure, tone letters, word boundaries, and contextual linguistic information.
📊 Training Data
The ASR model was fine-tuned using paired Cantonese speech and Liujgoj Romanized Cantonese transcription data.
Number of Training Examples: 39,774 audio–text pairs
Total Audio Duration: Approximately 26 hours of audio
Audio Format: 16 kHz, mono WAV
Transcription System: Liujgoj Cantonese Orthography (溜歌粵語羅馬字)
Data Source: Cantonese movies
Each training example consists of:
Cantonese Audio ↓ Liujgoj Cantonese Transcription
Training Data Summary
Item Details Training examples 39,774 Total audio duration ~18 hours Audio sampling rate 16 kHz Audio channels Mono Audio format WAV Speech language Cantonese Transcription target Liujgoj Cantonese Data source Cantonese movies Training Configuration Parameter Value Base model Qwen2.5-Omni-7B-SFT-merged-full Fine-tuning method LoRA LoRA target all Learning rate 1.0e-4 LR scheduler cosine Warmup steps 100 Epochs 3.0 Per-device batch size 2 Gradient accumulation steps 8 Effective batch size 16 Cutoff length 2048 Precision bfloat16 Final train loss 0.385 Total training time ~10.5 hours Key Features 🎙️ Direct Cantonese Speech Recognition Transcribes spoken Cantonese audio directly into Liujgoj Cantonese.
Data Source:
Yvthyvq/ZoengjYvthgaaij-MouxZaakhdungj-Liujgoj
Multi-film Cantonese dialogue corpus (cleaned and transliterated into Liujgoj orthography)
訓練相關命令：
train_asr_final.py
merge_asr_lora.py
batch_asr_1000.py
Example
Model Output
text Waahsvt hair cingjciux ge Hongjheij cojninx, dungjtinj ge mouxhor yathbouh, bakjfungj yvq douj, munq deih bingjsongj. Hair Gongjnaamx kaauganr hoirbinj yatj tiux daaihlouh zijsoengh, yauq yatj deoi cingjbingj, saurzaaph doujcoengj, ngaat zvh catj gaaj cauxcej, hoeng zvh bakjfongj haangxganr. Cinxminh gor saamj gaai caux cej aax, muiq gaaij zauh wanr zoekh yatj go naamxzir, douj haih svjsangj daarbaanh.
Reference Cantonese Text text
話説喺清朝嘅康熙初年，冬天嘅某一日，北風如刀，滿地冰霜啊。
喺江南靠近海邊一條大路之上，有一隊清兵，手執刀槍，押住七架囚車，向住北方行緊。
前便嗰三架囚車啊，每架就韞住一個男子，都係書生打扮。 As the model is still under development, transcription errors may occur, particularly in difficult acoustic conditions, rare vocabulary, proper nouns, and long-form speech.
大規模測試結果見：predictions_1000.jsonl
只有 5 條出現漢字：
- 第 310 行：
丛 - 第 391 行：
古 - 第 495 行：
油 - 第 697 行：
天 - 第 816 行：
孙子孙
註：測試語料：CanCLID/zoengjyutgaai （https://huggingface.co/datasets/CanCLID/zoengjyutgaai）中 lukdinggei 嘅前一千條語音。
Quickstart
Installation
pip install torch transformers librosa accelerate soundfile
Load the Model
python import torch from transformers import AutoProcessor from transformers.models.qwen2_5_omni.modeling_qwen2_5_omni import Qwen2_5OmniForConditionalGeneration
model_id = "Yvthyvq/Liujgoj-Cantonese-Qwen2.5-Omni-7B-ASR-V2"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) model.eval() Prepare Audio python import librosa
audio, sr = librosa.load("your_audio.wav", sr=16000) Run Inference python conversation = [ { "role": "user", "content": [ {"type": "audio", "audio": audio}, {"type": "text", "text": "聽呢段語音，用羅馬字寫出嚟。"}, ], } ]
text_prompt = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=False )
inputs = processor( text=text_prompt, audio=[audio], sampling_rate=16000, return_tensors="pt", ) inputs = {k: v.to(model.device) for k, v in inputs.items() if isinstance(v, torch.Tensor)}
with torch.no_grad(): generated_ids = model.thinker.generate(**inputs, max_new_tokens=256)
input_len = inputs["input_ids"].shape[1] generated_tokens = generated_ids[0][input_len:] transcription = processor.decode(generated_tokens, skip_special_tokens=True)
print(transcription)
Intended Use
This model is intended for research and development involving:
Cantonese Automatic Speech Recognition
Romanized Cantonese transcription
Liujgoj Cantonese Orthography
Cantonese speech corpus construction
Cantonese phonological research
Cantonese language learning
Speech-to-text pipelines
Cantonese TTS data preparation
Current Status
This model represents an experimental stage of the Liujgoj Cantonese multimodal model family.
The training pipeline has demonstrated that Liujgoj Cantonese capabilities can be developed through:
Original Qwen Tokenizer ↓ Liujgoj CPT ↓ Liujgoj SFT ↓ Liujgoj ASR
without pre-registering Cantonese syllables as additional tokenizer tokens.
Future improvements may include:
Larger Cantonese speech datasets
More diverse speakers
Additional acoustic domains
Long-form Cantonese transcription
Improved tone accuracy
Improved word segmentation
Automatic punctuation
Comparison with Whisper-based Cantonese ASR
Comparison with SenseVoice-based Cantonese ASR
Further integration with Liujgoj Cantonese TTS systems
Related Models
Liujgoj-Cantonese-Qwen2.5-Omni-7B-CPT-V2
Liujgoj-Cantonese-Qwen2.5-Omni-7B-SFT-V2.2
Liujgoj-Cantonese-Qwen2.5-Omni-7B-ASR-V2