CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged

This is an intermediate checkpoint, released as a baseline. It is a 30.5B-parameter mixture-of-experts model (3.3B activated) built from Qwen3-30B-A3B-Base by continual pre-training on 784M tokens of Cantonese and Hong Kong text, then adding the chat vector taken from Qwen3-30B-A3B-Thinking-2507.

It has Hong Kong knowledge and instruction following, acquired at zero training cost for the second of those. What it does not have is Cantonese reasoning: it thinks in Written Chinese. Installing that behaviour took three further stages, and the result is CantoneseLLM-v2.0-30B-A3B-Thinking.

Use this model if you want a Cantonese-knowledgeable instruct model and do not care what language the chain-of-thought is in, or you are reproducing the paper's ablations. Otherwise use the final model.

📄 Paper: CantoneseLLM v2: Reasoning in a Low-Resource Language (arXiv:2609.06970)

🧪 Evaluation Benchmark: hon9kon9ize/hkeval2025

What the chat vector does, and what it does not

The merge is one operation and requires no instruction data:

Δ_chat = θ_instruct − θ_base            # Qwen3-30B-A3B-Thinking-2507 − Qwen3-30B-A3B-Base
θ_merged = θ_cpt + Δ_chat               # added to the continually pre-trained checkpoint

It installs instruction following and alignment for free. It also carries over the donor's reasoning language, which continual pre-training does not override. On a translated GSM8K probe, under a system prompt that explicitly asks for Hong Kong Cantonese, this model opens its trace in Cantonese:

首先，用戶嘅問題係關於Janet嘅鴨蛋生意。我需要用香港廣東話回應，因為系統設定係CantoneseLLM。

(First, the user's question is about Janet's duck egg business. I need to respond in Hong Kong Cantonese, because the system is set to CantoneseLLM.)

…and then slides into Written Chinese later in the same span, ending up discussing the output language rather than thinking in it:

…現在，用香港廣東話回應。…在回應中，要用口語廣東話。

(…Now, respond in Hong Kong Cantonese. … In the response, colloquial Cantonese should be used.)

現在 jin6 zoi6 is the Written Chinese form where Cantonese uses 而家 ji4 gaa1, and 在⋯中 zoi6…zung1 is a Mandarin-shared construction. The final answer is correct and reads as Cantonese; the reasoning that produced it does not.

Trace length over eight fixed probes (mean tokens between the reasoning tags):

This model reasons at length — longer than its own donor. That is worth knowing: the length was never the problem, the language was.

Benchmark results

HKCanto-Eval (Cheng et al., 2025), reasoning mode on:

The merge costs 1.94 points against the donor (−2.55%), and the regression is concentrated in MMLU in English, which drops 5.94 points. Cultural is the one category that improves — 68.65 → 70.24 — which is the continual pre-training showing through, since that is the category most about Hong Kong.

This model scores 1.20 points above the final RLVR model. That is not a reason to prefer it. The benchmark is multiple-choice and does not measure the language of the chain-of-thought, which is the entire difference between the two checkpoints. Selecting on this average gets you a model that thinks in Written Chinese.

Artefacts released with this work

The CPT corpus itself is not released, but its two largest public constituents are.

Other models in this release

All four checkpoints are in the CantoneseLLM v2.0 collection.

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95)
print(tokenizer.decode(out[0][len(inputs.input_ids[0]):], skip_special_tokens=True))

vLLM

The stock command works — no special flags are required:

vllm serve hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged

Add --tensor-parallel-size N to shard across N GPUs, and --reasoning-parser qwen3 if you want the reasoning block returned separately as reasoning_content rather than inline in content (parser names vary by vLLM version).

OpenAI-compatible API

The served endpoint speaks the OpenAI protocol, so the official client works unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

resp = client.chat.completions.create(
    model="hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged",
    messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048,
)

msg = resp.choices[0].message
print(getattr(msg, "reasoning_content", None) or "")  # populated with --reasoning-parser
print(msg.content)

Or with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged",
    "messages": [
      {"role": "system", "content": "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"},
      {"role": "user", "content": "點解香港嘅雨季集中喺五月到九月？"}
    ],
    "temperature": 0.6, "top_p": 0.95, "max_tokens": 2048
  }'

Sampling: temperature 0.6, top-p 0.95 — the settings used for every evaluation reported above and in the paper. Avoid greedy decoding.

System prompt: the Cantonese system prompt above was used throughout evaluation. Behaviour with other system prompts, or with none, is not characterised.

Chat template and thinking mode are inherited from the donor. The chat vector came from Qwen3-30B-A3B-Thinking-2507, a reasoning-only model, so this checkpoint always produces a reasoning block. Do not expect /no_think to work as it does in the stock Qwen3 hybrid models.

Training pipeline

That is the whole pipeline for this checkpoint — no SFT, no DPO, no RLVR. The corpus was 32.9% Common Crawl (5.4% of the total in Cantonese), 20.9% English replay from Nemotron pre-training data, and 17.1% web fiction, which is the only source where Cantonese is sustained over tens of thousands of tokens.

The CPT learning rate was selected across eight settings from 1.0×10⁻⁵ to 5.0×10⁻⁵ by benchmark average rather than by training loss — the paper reports that loss anti-correlates with the quantity of interest in this regime.

Risks & Limitations

• It does not reason in Cantonese. This is the defining limitation and the reason the rest of the pipeline exists. The trace opens in Cantonese and drifts into Written Chinese. If you need Cantonese chain-of-thought, use the final model.
• Instruction following is inherited, not trained. It arrives through weight arithmetic from the donor and was never fine-tuned on Cantonese instruction data. Behaviour outside the donor's own distribution is not characterised.
• MMLU regression. 5.94 points below the donor in English MMLU. The merge is not free in capability terms even though it is free in compute terms.
• No translation, data-curation or judging ability. Those were installed at SFT, which this checkpoint precedes.
• CPT was 784M tokens — small by continual-pre-training standards (Taiwan-LLM used 35.1B, Swallow and SEA-LION 200B each). Hong Kong knowledge is real but thin, and the corpus leans on web fiction for long-form Cantonese.
• Alignment is whatever the chat vector carried over. No additional safety tuning was performed at any stage.
• Long context is inherited and untested — the CPT sequence length was 4,096 packed. Behaviour beyond that is whatever the Qwen3 base provides.
• Standard LLM caveats apply: it will hallucinate.

Citation

@misc{cantonesellm_v2,
      title={CantoneseLLM v2: Reasoning in a Low-Resource Language},
      author={Tsz Chung Cheng and Chung Shing Cheng and Chaak Ming Lau and Cheuk Hei Chong},
      year={2026},
      eprint={2609.06970},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.06970},
}

The chat vector method is due to Huang et al., Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages, ACL 2024, pp. 10943–10959.

Acknowledgements

Continual pre-training (CPT) was carried out on Cloud TPUs (Tensor Processing Units) from Google's TPU Research Cloud with MaxText. Post-training was carried out on computer resources offered under the category of General Projects by Research Institute for Information Technology, Kyushu University. Usage fee and cost of data-curation costs with proprietary APIs were covered by Votee AI

The RLVR stage builds on NVIDIA's NeMo-RL and NeMo-Gym, and on the Nemotron post-training and RL datasets.