LFM2.5-8B-A1B UltraData L2 49M
Continued-pretraining checkpoint based on
LiquidAI/LFM2.5-8B-A1B-Base.
Training
- Dataset:
openbmb/UltraData-Code - Stage: L2 continued pretraining
- Language: Python
- Context length: 8192
- Actual L2 input tokens: 48,990,537
- GPU: NVIDIA A100 80GB
- Precision: BF16
- LoRA rank: 32
- MoE expert LoRA rank: 1
- Peak learning rate: 5e-5
- Optimizer: AdamW
- Packing: enabled
Evaluation
| Benchmark | pass@1 |
|---|---|
| HumanEval | 0.482 |
| HumanEval+ | 0.439 |
Training performance
- L2 input tokens: 48.99M
- Runtime: ~2.14 hours
- Throughput: ~6.35K tokens/sec
Intended use
Experimental continued-pretraining checkpoint for coding specialization and further supervised fine-tuning.
Training data and licensing
This model was derived from LiquidAI/LFM2.5-8B-A1B-Base and trained
using openbmb/UltraData-Code.
Review the upstream LFM license and the dataset/source repository licenses before redistribution or commercial use.