CantoneseLLM-v2.0-8B-Thinking-Chat-Vector-Merged

This is an intermediate checkpoint, released as a baseline. It is an 8B dense model built from Qwen3-8B-Base by continual pre-training on 784M tokens of Cantonese and Hong Kong text, then adding the chat vector taken from Qwen3-8B.

It has Hong Kong knowledge and instruction following, acquired at zero training cost for the second of those — and it scores above its own donor on HKCanto-Eval, which the 30B-A3B merge did not manage. What it does not have is Cantonese reasoning: it thinks in Simplified Chinese. Installing that behaviour took three further stages, and the result is CantoneseLLM-v2.0-8B-Thinking.

Use this model if you want a Cantonese-knowledgeable 8B instruct model and do not care what language the chain-of-thought is in, or you are reproducing the paper's ablations. Otherwise use the final model.

📄 Paper: CantoneseLLM v2: Reasoning in a Low-Resource Language (arXiv:2609.06970)

🧪 Evaluation Benchmark: hon9kon9ize/hkeval2025

What the chat vector does, and what it does not

The merge is one operation and requires no instruction data:

Δ_chat = θ_instruct − θ_base            # Qwen3-8B − Qwen3-8B-Base
θ_merged = θ_cpt + Δ_chat               # added to the continually pre-trained checkpoint

It installs instruction following and alignment for free. It also carries over the donor's reasoning language and script, which continual pre-training on Traditional Chinese and Cantonese does not override. On a translated GSM8K probe, under a system prompt that explicitly asks for Hong Kong Cantonese, this model reasons entirely in Simplified Chinese:

嗯，用户问的是Janet每天赚多少钱。首先，我需要理清楚整个流程。题目说Janet的鸭子每天生16只蛋。

(Well, what the user is asking is how much Janet earns each day. First, I need to work through the whole process. The problem says Janet's ducks lay 16 eggs a day.)

The script leaks into the answer as well. Counting eggs, this checkpoint writes 只 where Hong Kong Cantonese requires the classifier 隻 zek3 — 只 can only be read zi2, which is a different word. On this single probe the paper counts six such classifier errors. The final RLVR model produces none.

Trace length over eight fixed probes (mean tokens between the reasoning tags):

This model reasons at roughly the donor's length. The length was never the problem; the language and the script were.

Benchmark results

HKCanto-Eval (Cheng et al., 2025), reasoning mode on:

The merge gains 2.15 points over the donor (+3.18%) — the only merge in this work that improved on the model it took the chat vector from. The gain is almost entirely Cultural, 57.94 → 66.67, an 8.73-point jump that is the continual pre-training showing through in the category most about Hong Kong. Linguistic also gains 2.00 and Academic & Professional 0.75, while English MMLU costs 1.04.

This model is the highest-scoring 8B checkpoint in the whole pipeline — 6.71 points above the final RLVR model. That is not a reason to prefer it. The benchmark is multiple-choice and does not measure the language of the chain-of-thought, which is the entire difference between the two checkpoints. Selecting on this average gets you a model that thinks in Simplified Chinese.

Artefacts released with this work

The CPT corpus itself is not released, but its two largest public constituents are.

Other models in this release

All four checkpoints are in the CantoneseLLM v2.0 collection.

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking-Chat-Vector-Merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95)
print(tokenizer.decode(out[0][len(inputs.input_ids[0]):], skip_special_tokens=True))

vLLM

The stock command works — no special flags are required:

vllm serve hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking-Chat-Vector-Merged

Add --tensor-parallel-size N to shard across N GPUs, and --reasoning-parser qwen3 if you want the reasoning block returned separately as reasoning_content rather than inline in content (parser names vary by vLLM version).

OpenAI-compatible API

The served endpoint speaks the OpenAI protocol, so the official client works unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

resp = client.chat.completions.create(
    model="hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking-Chat-Vector-Merged",
    messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048,
)

msg = resp.choices[0].message
print(getattr(msg, "reasoning_content", None) or "")  # populated with --reasoning-parser
print(msg.content)

Or with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking-Chat-Vector-Merged",
    "messages": [
      {"role": "system", "content": "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"},
      {"role": "user", "content": "點解香港嘅雨季集中喺五月到九月？"}
    ],
    "temperature": 0.6, "top_p": 0.95, "max_tokens": 2048
  }'

Sampling: temperature 0.6, top-p 0.95 — the settings used for every evaluation reported above and in the paper. Avoid greedy decoding.

System prompt: the Cantonese system prompt above was used throughout evaluation. Behaviour with other system prompts, or with none, is not characterised.

Chat template and thinking mode are inherited from the donor. The chat vector came from Qwen3-8B, a hybrid model, so this checkpoint keeps the stock Qwen3 thinking / non-thinking switch. All evaluation above was run with reasoning mode on.

Training pipeline

That is the whole pipeline for this checkpoint — no SFT, no DPO, no RLVR. The corpus was 32.9% Common Crawl (5.4% of the total in Cantonese), 20.9% English replay from Nemotron pre-training data, and 17.1% web fiction, which is the only source where Cantonese is sustained over tens of thousands of tokens.

At 8B the CPT ran at a single learning rate of 3.0×10⁻⁵, with 520, 530 and 540 steps compared to probe the effect of the final stretch of training. The 530-step checkpoint was selected.

Risks & Limitations

• It reasons in Simplified Chinese. This is the defining limitation and the reason the rest of the pipeline exists. The script leaks into answers through the 只 / 隻 conflation described above. If you need Cantonese chain-of-thought, or correct Traditional Chinese classifiers, use the final model.
• Instruction following is inherited, not trained. It arrives through weight arithmetic from the donor and was never fine-tuned on Cantonese instruction data. Behaviour outside the donor's own distribution is not characterised.
• No translation, data-curation or judging ability. Those were installed at SFT, which this checkpoint precedes.
• CPT was 784M tokens — small by continual-pre-training standards (Taiwan-LLM used 35.1B, Swallow and SEA-LION 200B each). Hong Kong knowledge is real but thin, and the corpus leans on web fiction for long-form Cantonese.
• Alignment is whatever the chat vector carried over. No additional safety tuning was performed at any stage.
• Long context is inherited and untested — the CPT sequence length was 4,096 packed. Behaviour beyond that is whatever the Qwen3 base provides.
• Standard LLM caveats apply: it will hallucinate.

Citation

@misc{cantonesellm_v2,
      title={CantoneseLLM v2: Reasoning in a Low-Resource Language},
      author={Tsz Chung Cheng and Chung Shing Cheng and Chaak Ming Lau and Cheuk Hei Chong},
      year={2026},
      eprint={2609.06970},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.06970},
}

The chat vector method is due to Huang et al., Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages, ACL 2024, pp. 10943–10959.

Acknowledgements

Continual pre-training (CPT) was carried out on Cloud TPUs (Tensor Processing Units) from Google's TPU Research Cloud with MaxText. Post-training was carried out on computer resources offered under the category of General Projects by Research Institute for Information Technology, Kyushu University. Usage fee and cost of data-curation costs with proprietary APIs were covered by Votee AI

The RLVR stage builds on NVIDIA's NeMo-RL and NeMo-Gym, and on the Nemotron post-training and RL datasets.