🌿 Gan Cao v3.2
A 229M Parameter Bilingual TCM Model | MASSIF Accelerator Class
Overview
Gan Cao v3.2 is a 229M parameter bilingual language model fine-tuned for Traditional Chinese Medicine (TCM) and Eastern philosophy. It is built on the MASSIF architecture (Pre-LN + RMSNorm + SwiGLU + RoPE) with a Qwen tokenizer for proper Chinese character handling.
Unlike earlier versions that used GPT-2 tokenizer (which fragmented Chinese characters), v3.2 uses Qwen's tokenizer for accurate Chinese text representation.
Key Improvements from v3
| Aspect | v3 | v3.2 |
|---|---|---|
| Tokenizer | GPT-2 (fragments Chinese) | Qwen (proper Chinese tokens) |
| Parameters | 63.5M | 229M |
| d_model | 512 | 768 |
| n_layers | 12 | 16 |
| n_heads | 8 | 12 |
| τ_eff | 0.58 (Neutral) | 0.68 (Accelerator) |
| Training Data | 85 bilingual pairs | 6 streaming datasets (10k steps) |
MASSIF Classification
Under the MASSIF taxonomy (Solis, 2026), Gan Cao v3.2 achieves Accelerator Class:
| Metric | Value | Threshold |
|---|---|---|
| τ_eff | 0.68 | > 0.55 (Accelerator) |
| Loss Drop | 109 → 38 | -65% |
| Architecture | Pre-LN + RMSNorm + SwiGLU + RoPE | Validated |
Evolution Across Versions
| Model | Params | τ_eff | Class |
|---|---|---|---|
| Gan Cao v1 | 63.5M | 0.00 | Neutral |
| Gan Cao v2 | 63.5M | 0.58 | Neutral |
| Gan Cao v3 | 63.5M | 0.93* | Accelerator* |
| Gan Cao v3.2 | 229M | 0.68 | Accelerator |
*Note: v3 measurement used conversational prompts; v3.2 uses N=50 stress protocol for comparability with taxonomy.
Architecture
| Component | Value |
|---|---|
| d_model | 768 |
| n_layers | 16 |
| n_heads | 12 |
| d_head | 64 |
| Vocab Size | 151,643 (Qwen) |
| Context Length | 512 |
| Total Parameters | 229M |
| Architecture | Pre-LN + RMSNorm + SwiGLU + RoPE |
| Position Encoding | RoPE (Rotary Position Embedding) |
| Activation | SwiGLU |
Training Data
Gan Cao v3.2 was trained on 6 streaming datasets (10,000 steps, 8-hour training on NVIDIA T4):
| Dataset | Weight | Content |
|---|---|---|
| TCM Pretrain (ShizhenGPT) | 25% | 5B tokens TCM medical texts |
| Classical Chinese (ctext) | 20% | Pre-Qin philosophy, classical grammar |
| Buddhist Canon | 15% | Chinese Buddhist scriptures |
| Ancient Documents (ByteDance) | 15% | Classical → Modern translations |
| TCM Chat | 15% | Clinical TCM conversations |
| Chinese Poetry | 10% | Tang poems (Daoist/Buddhist themes) |
Total training composition: 90% medical corpora, 10% philosophical context
Usage
Installation
pip install transformers torch