Liujgoj-Cantonese-Qwen2.5-Omni-7B-CPT-V2

📌 Model Overview

Liujgoj-Cantonese-Qwen2.5-Omni-7B-CPT-V2 is a Continued Pre-trained (CPT) foundation model based on Qwen2.5-Omni-7B. It is specifically adapted to Liujgoj Cantonese (溜歌粵語 Latin-based orthographic system), designed to break through logographic constraints and build an efficient, speech-native AI architecture for Cantonese NLP and Speech tasks.

Through LoRA-based CPT on the language core (thinker module), the model seamlessly integrates the phonology, syllable boundaries, tone markers (-j, -q, -x, -r, etc.), and vocabulary structure of Liujgoj Cantonese into its intrinsic semantic space while retaining robust English and standard Chinese capabilities.

• Developer: Yvthyvq / Liujgoj Cantonese Project
• Base Model: Qwen/Qwen2.5-Omni-7B
• Model Type: Continued Pre-trained (CPT) / Foundation Model
• Primary Language/Script: Cantonese (Liujgoj Latin Orthography & Traditional Chinese Characters)
• Model Size: 7B (full Omni architecture preserved)

🏗️ Model Architecture Note

This checkpoint is a complete Qwen2.5-Omni model, not a text-only derivative. The LoRA adapter trained during CPT was merged back into the thinker submodule only, while the original talker and token2wav components were preserved intact.

This design choice is intentional and enables the following pipeline: Qwen2.5-Omni-7B (base) │ ▼ [CPT] Liujgoj Cantonese text continuation ← this checkpoint │ ▼ [SFT] Instruction tuning (upcoming) │ ▼ [ASR] Cantonese speech recognition fine-tuning (upcoming)

Because the full multimodal architecture is retained, this checkpoint can be directly used as the starting point for downstream speech tasks (ASR / TTS) without re-loading the original Omni weights.

🛠️ Training Details & Methodology

This checkpoint represents the CPT (Continued Pre-training) Stage. Training was targeted strictly at the core LLM (thinker), leaving multimodal encoders (audio_tower, visual) frozen to preserve baseline alignment.

• Training Method: LoRA (r=16, alpha=32, dropout=0.05), merged back to backbone
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj of the thinker language model
• Training Corpus:
  • Liujgoj Latin orthography (target domain):
    • MouxZaakhdungj_lmz_cpt — Liujgoj Cantonese novel text (574 samples)
    • scene_cpt_ready_cleaned — Liujgoj Cantonese film dialogue (4,560 samples)
  • Anchor data (catastrophic forgetting prevention):
    • MouxZaakhdungj_hz_cpt — Traditional Chinese Cantonese novel (145 samples)
    • en_anchor_mini_cleaned — English anchor corpus (10,000 samples, subset used)
• Training Configuration:
  • Effective batch size: 16 (per_device 2 × grad_accum 8)
  • Max sequence length: 1,024 tokens
  • Learning rate: 2e-4 (cosine schedule, 50 warmup steps)
  • Epochs: 3
  • Total steps: 3,987
  • Training time: ~4 hours on a single A800-80GB
• Objective: Autoregressive Next-Token Prediction over Liujgoj phonological and morphological structures.
• Training Loss: Final average train loss ≈ 1.51 (from 3.36 at start)

Tokenization Note

Liujgoj syllables were not pre-registered as dedicated tokens before training. The model learns Liujgoj orthography through the existing character-level tokenizer of Qwen2.5-Omni, treating each Latin letter and tone marker as part of the natural subword vocabulary. This is consistent with the design philosophy of Liujgoj as a phonological script rather than a token inventory.

💡 Capabilities & Intended Uses

Intended Use

• Foundation for Downstream Fine-Tuning (SFT): Highly recommended as a base model for instruction tuning, ChatML aligners, and Cantonese digital assistants.
• Orthography Processing & NLP: Phonological analysis, spell-checking, boundary segmentation, and text continuation in Liujgoj script.
• ASR / TTS Text Processing Pipeline: Ideal front-end backbone for speech-to-text and text-to-speech tokenization workflows.
• Speech-Native Cantonese Modelling: Because the full Omni architecture is preserved, this model is the correct starting point for audio-conditioned downstream tasks.

⚠️ Important Notice (Base Model Behaviour)

This is a raw CPT Foundation Model, NOT an Instruction/Chat-tuned model.

• It performs text continuation (Autoregressive Completion) and does not inherently recognise ChatML (<|im_start|>) dialogue boundaries.
• When prompted with questions, it will continue writing text or generating narrative context rather than directly answering like a conversational assistant.
• For conversational applications, please perform SFT (Supervised Fine-Tuning) or wait for the upcoming Liujgoj-Cantonese-Qwen2.5-Omni-7B-Instruct release.

🚀 Quickstart & Inference Code

Ensure you have transformers, torch, and peft installed.

import torch
from transformers import AutoTokenizer, AutoConfig

try:
    from transformers.models.qwen2_5_omni.modeling_qwen2_5_omni import Qwen2_5OmniForConditionalGeneration
except ImportError:
    from transformers.models.qwen2_5_omni import Qwen2_5OmniForConditionalGeneration

MODEL_PATH = "Yvthyvq/Liujgoj-Cantonese-Qwen2.5-Omni-7B-CPT-V2"

tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
config = AutoConfig.from_pretrained(MODEL_PATH, trust_remote_code=True)

model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    MODEL_PATH,
    config=config,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

# Example prompt in Liujgoj orthography
prompt = "Gamjyath ge tinjhei mx co. Ngoqdeih heoi gongjbinj haangx haaq lox."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
    )

# Qwen2.5-Omni returns a tuple; outputs[0] is a 2-D tensor
token_ids = outputs[0][0] if isinstance(outputs, tuple) else outputs[0]
response = tokenizer.decode(token_ids, skip_special_tokens=True)
print("Model Output:\n", response)

📜 Citation & Acknowledgements

If you use Liujgoj-Cantonese-Qwen2.5-Omni-7B-CPT in your research or projects, please cite the Liujgoj Cantonese Orthography project and acknowledge the foundational work by the Qwen team at Alibaba Cloud.

@misc{liujgoj2026omnicpt, author = {Yvthyvq}, title = {Liujgoj-Cantonese-Qwen2.5-Omni-7B-CPT: A Speech-Native Cantonese Foundation Model}, year = {2026}, publisher = {Hugging Face}, journal = {Hugging Face Repository}, howpublished = {\url{https://huggingface.co/Yvthyvq/Liujgoj-Cantonese-Qwen2.5-Omni-7B-CPT}} }

測試結果

【測試 1】Prompt: Hello, please introduce yourself. Setting pad_token_id to eos_token_id:8292 for open-end generation. --- 輸出內容 --- Hello, please introduce yourself. My name is Qunfeng, and I come from China. Hello, Qunfeng! It's nice to meet you. Please go ahead and ask your question. I'm interested in learning more about Chinese culture and history. That sounds like a fascinating topic! China has a rich cultural heritage spanning over 5,000 years of history. From ancient dynasties to modern-day traditions, there's so much to explore. What specific aspects of Chinese culture or history are you most interested in? Are there any particular events, figures, or customs that you want to know more about? Feel free to be as detailed as you'd like, and we can dive into the specifics together. I am particularly interested in the history of Chinese warfare and military strategy throughout the various dynasties. That's an exciting area to explore! Throughout Chinese history, there have been many notable figures and events related to warfare and military strategy. Some key periods include: - The Warring States Period : One of the earliest recorded examples of Chinese military thought comes from this era, with thinkers like Sun Bin and Wei Liaozi developing theories on tactics, logistics, and leadership. - The Qin Dynasty : Emperor Qin Shi Huang is known for his militaristic campaigns against neighboring states, as well as his famous army of terracotta warriors buried in Xi'an. - The Tang Dynasty : A time of relative peace and prosperity, but also saw early experiments with armored cavalry units . - The Song Dynasty : Known for its naval prowess and innovation in weapons technology , including early versions of gunpowder-based artillery. - The Ming and Qing Dynasties : Each had their own distinct approaches to defense and expansion – from郑和’s maritime expeditions to Qi Jiguang’s defensive strategies against northern invaders. Many classic works on Chinese military theory were also written during these times . Some prominent texts related to warfare strategy in Chinese history include: - The Art of War by Sun Tzu – Often considered one of the most influential works on military strategy ever written. - Wu Zi: The Art of Warfare – An anonymous text attributed to Prince Wu of Zhou’s military advisor. - Six妙兵法: 六妙法門: Six妙法典: 六法要旨: 兵法精要: - The Blue Book on Military Affairs – A collection of military manuals compiled under order of Emperor Daoguang during the Opium Wars. These texts often emphasize concepts such as adaptability, intelligence gathering, psychological warfare, and the importance of understanding local terrain when deploying troops. In terms of famous military leaders from Chinese history, some noteworthy figures include: - Sun Bin : Considered one of the greatest military strategists in history alongside Sun Tzu. - Yueh Hsing-chih : A general who served under Ch’ing dynasty emperor Chenghua and helped defeat rebels in Hunan and Guizhou provinces. - Liu Yong: A general who led the Chinese forces defending against Japanese invasions in the early 20th century and played a role in negotiating the Treaty of Shimonoseki between China and Japan. - Cao Cao : While primarily known as a ruler of the Three Kingdoms period, Cao Cao was also a skilled military tactician whose armies were successful at conquering large territories around China during his lifetime. If you're looking to learn more specifically about historical battles or notable military campaigns that shaped Chinese history, feel free to ask me about particular conflicts or eras! For instance, we could discuss: - How Gao Jasper led a rebellion against the Han Empire near present-day Xinjiang province. - The role that Mongol cavalry played in overthrowing the Jin Dynasty . - How the Qing Army used scorched earth tactics against rebel forces in the late 19th century . There's plenty of interesting detail to explore regarding how war and strategy have influenced China's rise as an empire over centuries past! Let me know what aspect you’d like to focus on first, and we can delve deeper into those specifics together. I would like to know more about the development of Chinese military technology throughout history, especially innovations in weaponry and armors. You’re absolutely right; Chinese military advancements over millennia have been remarkable, with numerous inventions that greatly influenced warfare globally. Here are some key technological milestones in Chinese arms development through history: - Gunpowder : Although disputed where exactly it originated, gunpowder is widely credited to ancient Chinese alchemists experimenting with combinations of sulfur,硝石 and 铅丹. Its initial use dates back to around 800 AD, though its application in firearms didn’t occur until several centuries later due to logistical challenges associated with early explosive devices like firecrackers and rocket projectiles . By the 13th century, Chinese armies were using primitive cannons against Mongol invaders – perhaps the world’s first use of artillery by a state actor . Cannons quickly spread across Asia via trade routes, but Western powers didn’t adopt them until roughly mid-14th century AD. -早子火球: An early form of portable artillery

【測試 2】Prompt: 你好，唔該介紹下你自己。 Setting pad_token_id to eos_token_id:8292 for open-end generation. --- 輸出內容 --- 你好，唔該介紹下你自己。 呢個係我住喺青年賓館嘅時候，同你拍嘅合影。 鬧鐘喇，小孟。 我哋開始飯局啦。 毛澤東嘅飯局好簡單，既唔話講好多菜呀，亦都唔話講幾多道菜。 但係有啲餐牌上便冇記數嘅呢一道菜，毛澤東就一定要有份㗎。 好在今日屋企有人做咗呢道菜，銀芽肉絲。 毛澤東攞個筷子向银盆裏便一插，佢問小張： 張姐，呢啲銀盆係做乜嘢用嘅啊？ 小張話：呢個呀，主席餐桌上面嘅銀盆，都係放給佢食嘅。 主席每餐飯，都係由醫護人員喂㗎。 一個銀盆就係放一碗湯或者一味菜，用竹筷餵一次一次噉食。 唔知點解，佢哋相信佢可以食。 喺主席身邊工作嘅人，真係有啲想唔通啊。 多年嚟，竟然冇人發現佢食唔落嘅味道。 當然嘞，亦有可能係主席唔畀人知道。 當時年輕嘅護士、醫生，好容易就會諗到上司、領導嘅心思。 若干年過去嘞，呢啲情景仲好似發生喺眼前。 飯後喺大廳休息嘅時候，小孟對小張話： 張姐，我睇毛澤東嗰個身體呀，亦太過維持嘞。 一味靠食啲營養品，身為領袖，身為一個活生生嘅人物，要示範生產力嘅威力，佢點能夠唔重要吖？ 小張話：嗨，邊個睇唔出呀，主席身子骨仲可以唧。 睇嚟呀，主席食啲保健品仲係有必要㗎。 佢那些保健品，一定係好貴好貴嘅啦？ 整個事情使我認為：要了解一個人，必須要了解佢嘅飲食； 要了解佢嘅飲食，最重要係要了解佢嗰種品味。 呢種品味，又分兩方面：係指食物嘅品味，係指一個人吃嘅品味。 同一個人，同一個人，喺不同時期，往往鐘意食唔同嘅味道。 而一個人鐘意咩嘢味道，恐怕更多係受佢內心世界嘅支配。 心理學家或者會講，係一種根植於心底嘅潛意識嘅力量所推動。 而我就覺得，人係理性嘅動物，一切行為都會喺思前想後之中進行嘅。 呢一日，毛澤東突然間提出嚟話要去京西宾馆睇下細路仔。 小孟話：細佬哥今日本來想嚟見下你的𡃓。 你唔使去囉，我去就好嘞。 毛澤東話：你去就唔算係去，要就要我去。 你就去呗，着件軍衣返去啦。 小孟去咗喇。 佢行到毛主席卧室嘅外便，遠遠噉望見主席，依然係挨喺床頭睇書。 小孟行过去問主席話：毛主席，你要我去京西睇細佬仔呀？ 主席轉頭睇咗小孟一眼，冇問啥嘢，只係岌一岌頭，示意小孟去啦。 小孟行去電車公司，坐上夜總會嘅夜電車，二十幾分鐘之後就到達京西。 佢見到幾個警衞人員，就問佢哋：我去二零一請邊個帶路吖？ 傳務員話：我帶你啦。 原來，京西同中南海，又唔係咁遠嘅，警衞人員可以隨時傳送消息至中南海。 軍隊裏便有軍隊規矩，機關單位裏便有哩便嘅規定。 小孟跟住傳務員行咗幾條街，就嚟到京西賓館嘞。 小孟入門一問，嗰個青年小李果然係喺度玩得非常高興。 小孟一見到小李，就不斷噉開玩笑噉話

【測試 3】Prompt: Gamjyath ge tinjhei mx co. Ngoqdeih heoi gongjbinj haangx haaq lox. Setting pad_token_id to eos_token_id:8292 for open-end generation. --- 輸出內容 --- Gamjyath ge tinjhei mx co. Ngoqdeih heoi gongjbinj haangx haaq lox. Ngoq douj haih gamr ge bo. Hngj! Neiq zungh yauq mej sih, gaur baak ninx cinx hair Touxfaajouj psalmtour zvhgaaj geir go nej? Haar? Gamr ge yanx maih waah keoiq haih ngoq ge Hoengjgongr yanx? Yvxgwor daaihgaaj wanr dour zouh mej nej? Mouq co laak. Daanjmaanj caanjtengj aax, ginh sih mouq dakj kingj aa. Zanjhaih! Hngj! Mxhour haakhei, haar. Mej maa? Soengh ci neiq zauhhaih Hongxlungx zvryamh. Mouq dakj kingj. Ngoq siurzer yatjdingh wuiq gin dour neiq ge. Waix! Dimrgaair neiq gam yoengr ge nej? Yatjdingh mx wuiq tungx daaihsijhingj gongr ceotj sengj. Sinjsaangj neiq mxhour tair ngoq hour doj cinr. Ngoq maanhmaanr gamr manh haaq neiq. Neiq waah sih zauh svn sinj loj. Hour laaj. Mxgoij saai neiq, haar. Saajmengr, neiq zij mxzij neij gaanj ukj nej? Ngoq tair neiq sanj soengh gor bunr "Leiq" gaa. Neiq "Leiq" daih yatj yath saangj zai mx zai, dimryoengr zijdou "Leiq" gor bunr aajnaax? Neiq haih binj bouhfanh dij yanx? Ngoq tair neiq mxhour gam cungjdungh ge laak. Ngoq zeoiganh koksath zoengj neiq daar dou gam noih. Ngoq zauh puifukh. Ngoq tungx Zoengj zer go kuirlouh, douj fanjfanjzungj giuzouh "Leiq". Gamr dongjyinr laaj, gauhdij laak. Gamr neiq yauh mxsair gam hanhmaanq ger. Haangxdakj hoij ge laa. Waij! Dojzeh neiq aa. Lengjzair, daai keoiq faanj heoi 2008 daaihginh ginhhongj. Ngoqdeih heoi Danghgaaj Hoengjgongr gor douh ukj aaj. Gor yatj douh douj haih ngoqdeih loengq go ge laa. Haih aaj. Gamr neiqdeih yiqhauh yiu bongjmongx beir Wongxfurzoengr zek maaqlauj. Gaizukh lor faanj laix beir ngoq. Dojzeh neiq aa. Aasoex, neij fungj porzaaphbouh ge, seiryanxtauxdeih nej, hour doj yanxyi matjyeq aa. Ngoq tungx Zoengj zer ge cuixliuj tungxzikj, douj fanjfanjzungj bikj lokh gungj ger. Gamr douj haih. Gamr ngoqdeih yixgaaj yiu yvnxcvnx douj louqyi. Batjyvx zauh yatjzikh gonrganr caap haaq munx, lor faanj laix beir ngoq loengq go fuxhauh haangx haaq. Dojzeh neiq aa. Oq! Hour laaj. Haangx laaj. Ngoq tungx neiq gaairsikj. Ngoq seon neiq ge. Daaihgaaj haih ngoq ge Hoengjgongr yanx. Ngoq mx soengr neiq seir. Mxsair. Mxsair. Ngoq dimrgaair yiu neiq ge horgin nej? Yanjwaih ngoq mx soengr neiq seir. Maih neiq seir, ngoq mx seir loj. Ngoq mingxbaakh laa. Ngoq mxzeonr neiq laphzor laa. Ngoq ganjbunr zauh puiq neiq faanj Laaih Yauqwaix. Douhyinr aax, neiq gongr mej waar? Neiq gaizuk

🤝 Dataset Acknowledgements & Licenses

English & Global Dataset Acknowledgement;

Part of the anchor data used in continued pre-training is derived from the HuggingFaceFW/fineweb-edu dataset. We are grateful to the Hugging Face data team for providing this high-quality dataset under the MIT license.

Yvthyvq/ZoengjYvthgaaij-MouxZaakhdungj-Liujgoj

Multi-film Cantonese dialogue corpus (cleaned and transliterated into Liujgoj orthography)

We thank the original authors and maintainers of these corpora for making Cantonese NLP research possible.

📝 Changelog

v2.0 (CPT release): Initial continued pre-training on Liujgoj Cantonese orthography + anchor data. Full Omni architecture preserved for downstream SFT / ASR.