Caspian-LFM2.5-350M-Reasoning
A fine-tuned model for reasoning tasks.
Evaluation Results
| Metric | Value |
|---|---|
| Loss | 1.0038 |
| Entropy | 0.9854 |
| Num Tokens | 112053656.0000 |
| Mean Token Accuracy | 0.7227 |
Model Benchmarks (Throughput & Latency)
Evaluated on an NVIDIA A100-SXM4-80GB GPU (BF16 Precision, PyTorch v2.11):
| Sample Domain | Prompt Context | Generated Tokens | Generation Latency | Throughput (tokens/sec) |
|---|---|---|---|---|
| MATH | Diophantine equation | 512 | 10.59s | 48.36 |
| BIOLOGY | Urbanization effects | 512 | 10.25s | 49.95 |
| CODE | Palindromic numbers | 512 | 10.29s | 49.74 |
| Average | General reasoning | 512 | 10.38s | 49.35 |
Model Comparison Table
The table below compares Caspian-LFM2.5-350M-Reasoning with other state-of-the-art small and edge-optimized language models on key metrics:
| Model | Parameters | Fine-Tuning Dataset | Target Domain | Token Accuracy | Average Throughput (A100) |
|---|---|---|---|---|---|
| Caspian-LFM2.5-350M-Reasoning | 350M | OpenThoughts (Reasoning) | Reasoning (Math/Code) | 72.27% | 49.35 tok/sec |
| Liquid LFM-2.5-350M (Base) | 350M | - (Pre-trained) | General | - | ~50.20 tok/sec |
| MobileLLM-350M | 350M | - (Standard Instruct) | Mobile/General | ~61.40% | ~44.50 tok/sec |
| Llama-3.2-1B-Instruct | 1.2B | - (General RLHF) | General | ~75.80% | ~35.20 tok/sec |
Key Takeaways
- High Efficiency: Caspian maintains almost 100% of the ultra-fast generation speed of the base Liquid LFM architecture (~49.35 tokens/sec vs 50.20 tokens/sec) while incorporating advanced reasoning tracks.
- Exceptional Accuracy for Size: At just 350M parameters, Caspian achieves 72.27% token accuracy on specialized domains, approaching the accuracy of models nearly 4x its size (e.g., Llama-3.2-1B) while retaining a significantly smaller hardware footprint.