PhoneLM-0.5B is a 0.5 billion parameter decoder-only language model pre-trained on 1.1 trillion tokens.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = 'mllmTeam/PhoneLM-0.5B'
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)
inp = tokenizer("Machine Learning is ", return_tensors="pt")
inp = {k: v.to('cuda') for k, v in inp.items()}
out = model.generate(**inp,
max_length=256,
do_sample=True,
temperature=0.7,
top_p=0.7
)
text = tokenizer.decode(out[0], skip_special_tokens=True)
print(text)
Model Details
- Developed by: mllmTeam
- Model type:
PhoneLM 0.5Bmodels are auto-regressive language models based on the transformer decoder architecture. - Language(s): English
- Paper: PhoneLM Technical Report
- Library: PhoneLM
Model Architecture
The model is a decoder-only transformer architecture with the following modifications:
| Hidden Size | Layers | Heads | Sequence Length |
|---|---|---|---|
| 1024 | 24 | 16 | 2048 |
- Position Embeddings: Rotary Position Embeddings (Su et al., 2021) applied to the first 25% of head embedding dimensions for improved throughput following Black et al. (2022). PhoneLM quantized the sin and cos values in Rotary Position Embeddings to 8-bit integers.
- Normalization: LayerNorm (Ba et al., 2016) with learned bias terms as opposed to RMSNorm (Zhang & Sennrich, 2019).
- Biases: We remove all bias terms from the feed-forward networks and multi-head self-attention layers, except for the biases of the query, key, and value projections (Bai et al., 2023).
- ReLU Activation Function: ReLU(Glorot et al., 2011) activation functions are adopted in feed-forward networks.
- Tokenizer: We use the SmolLM(Allal et al., 2024)'s tokenizer with a vocabulary size of 49,152.
Training Dataset
The training dataset PhoneLM used is comprised of a filtered mixture of open-source large-scale datasets available on the HuggingFace Hub: DCLM-baseline(Li et al., 2024), StarCoder (Li et al., 2023), OpenWebMath (Paster et al., 2023) and Dolma (Soldaini et al., 2024).
Evaluation Results
| Model | HellaSwag | WinoGrande | PIQA | SciQ | BoolQ | ARC Easy | ARC Challenge | Average |
|---|---|---|---|---|---|---|---|---|
| PhoneLM-0.5B | 54.0 | 57.9 | 73.3 | 85.1 | 60.7 | 60.4 | 31.6 | 60.43 |
| Pythia-410M | 40.6 | 53.7 | 66.9 | 72.4 | 60.3 | 45.9 | 24.5 | 52.04 |
| OPT-350M | 36.8 | 52.3 | 64.3 | 68.5 | 57.6 | 40.1 | 23.7 | 49.04 |
| BLOOM-560M | 36.9 | 51.7 | 65.0 | 71.7 | 53.3 | 41.8 | 23.7 | 49.16 |
| MobiLlama-500M | 51.1 | 53.4 | 70.9 | 76.4 | 55.7 | 46.0 | 26.6 | 54.30 |
| OpenELM-450M | 54.0 | 58.0 | 72.3 | 79.4 | 55.8 | 48.1 | 27.6 | 56.46 |
| SmolLM-360M | 53.5 | 56.8 | 71.5 | 84.2 | 55.4 | 63.8 | 36.0 | 60.17 |
| Qwen 1.5-500M | 49.2 | 55.7 | 69.5 | 82.5 | 49.5 | 52.3 | 29.4 | 55.44 |
| Cerebras-GPT-590M | 32.3 | 49.8 | 62.8 | 68.2 | 59.2 | 41.2 | 23.5 | 48.14 |
License
- This repository is released under the Apache-2.0 License.
Citation
@misc{yi2024phonelmanefficientcapablesmall,
title={PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training},
author={Rongjie Yi and Xiang Li and Weikai Xie and Zhenyan Lu and Chenghua Wang and Ao Zhou and Shangguang Wang and Xiwen Zhang and Mengwei Xu},
year={2024},
eprint={2411.05046},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2411.05046},
}