One-Bit BLM Chimera — Causal Fix
Byte-level Language Model + Mixture of Experts + 1-bit BitNet, distilled dari Qwen/Qwen2.5-14B.
Training dijalankan dengan 2 GPU terpisah: satu khusus menjalankan teacher (inference only), satu lagi khusus training student (forward, backward, optimizer) -- bukan replikasi data-parallel. Load-balancing loss ditambahkan untuk mencegah expert collapse di MoE.
Checkpoint ini: pytorch_model_epoch3.pth (epoch 3 dari 50 rencana total).
File pytorch_model.pth selalu berisi salinan checkpoint terbaru; checkpoint
per-epoch tersedia terpisah dengan akhiran _epochN.pth di repo ini untuk
membandingkan progres antar epoch.
Quick Start
from huggingface_hub import hf_hub_download
exec(open(hf_hub_download("Yosua69/one_bit_blm_chimera_causal_fix", "inference.py")).read())
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = load_from_hub(device=device)
print(generate(model, device, prompt="The history of", max_bytes=150))
Arsitektur
| Komponen | Detail |
|---|---|
| Input | Raw bytes (vocab: 257, termasuk EOS) |
| Sequence mixing | GRU multi-layer, causal |
| Core | 4× blok MoE per-posisi (4 experts masing-masing) |
| Linear | 1-bit AbsMean quantization (BitNet) |
| Generation | Incremental, hidden state di-cache |
| Teacher | Qwen/Qwen2.5-14B (4-bit, knowledge distillation) |
| Load balancing | Importance loss (Shazeer et al.) di tiap layer MoE |