Xmodel-LM-1.1B
1. Model Summary
Xmodel-LM is a 1.1-billion-parameter causal language model pretrained from scratch by XiaoduoAI on ~2 trillion tokens of bilingual (Chinese & English) data.
Despite its compact size, it outperforms many open-source 1B-scale models (e.g. TinyLlama-1.1B, OPT-1.3B, Pythia-1.4B) on both English and Chinese benchmarks.
Paper: arXiv 2406.02856
Github: XiaoduoAILab/XmodelLM
Official checkpoints: XiaoduoAILab/Xmodel-LM
2. Intended Uses
| Task | Zero-shot | Fine-tune | Notes |
|---|---|---|---|
| Chinese & English text generation | ✅ | ✅ | Dialog, QA, copy-writing |
| Commonsense reasoning (ARC, HellaSwag, Winogrande …) | ✅ | — | Already evaluated in paper |
| Math word problems (GSM8k) | ⚠️ | ✅ | 2.58% 5-shot, better after ft |
| Code generation | ⚠️ | ✅ | Contains 11% code tokens |
Primary audience: researchers & practitioners who need a small, fast, bilingual base model for downstream adaptation or edge deployment.
3. How to Use
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained("XiaoduoAILab/Xmodel-LM")
model = AutoModelForCausalLM.from_pretrained("XiaoduoAILab/Xmodel-LM", torch_dtype=torch.float16).to(device)
prompt = "中国的四大发明是"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=128, do_sample=True, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))
4. Training Details
| Hyper-parameter | Value |
|---|---|
| Parameters | 1.1 B |
| Layers | 24 |
| Hidden size | 2048 |
| Attention heads | 32 (GQA 4 KV-heads) |
| Context length | 4096 |
| Vocabulary | 32 k (SentencePiece unigram) |
| Training tokens | 2.0 T |
| Warmup steps | 2 k |
| Peak lr | 6e-4 |
| Schedule | cosine → 6e-5 |
| Batch size | ~3.5 M tokens |
| Optimizer | AdamW (β1=0.9, β2=0.95) |
| Weight decay | 0.1 |
| Precision | bfloat16 |
| Hardware | 7×H800, DDP |
| MFU | ≈55% |
5. Evaluation Highlights (zero-shot unless noted)
| Benchmark | Xmodel-LM-1.1B | TinyLlama-1.1B | Qwen1.5-1.8B |
|---|---|---|---|
| ARC-e | 62.29 | 60.31 | 64.69 |
| HellaSwag | 45.96 | 44.98 | 45.49 |
| BBH (3-shot) | 27.34 | 26.75 | 13.84 |
| GSM8K (5-shot) | 2.58 | 1.97 | 33.59 |
| ARC-zh | 26.24 | 21.37 | 32.14 |
| XCOPA-zh | 60.60 | 56.80 | 66.00 |
Full tables in paper §3.
6. Limitations & Bias
- Math & code: still weak on GSM8k and coding benchmarks; recommend supervised fine-tuning for arithmetic or programming tasks.
- Knowledge cutoff: pretraining data up to ~2024-03; no live Internet access.
- Safety: not aligned with RLHF; may produce harmful or biased content — apply moderation before production use.
- Chinese ratio: only ~15% tokens; while better than most 1B models, still inferior to native Chinese models on classical or domain-specific text.
7. Citation
If you use Xmodel-LM in your research, please cite:
@misc{xmodel-lm-2024,
title={Xmodel-LM Technical Report},
author={Wang, Yichuan and Liu, Yang and Yan, Yu and Wang, Qun and Huang, Xucheng and Jiang, Ling},
eprint={arXiv:2406.02856},
url={https://arxiv.org/pdf/2406.02856},
year={2024}
}