Droplet-v2
Droplet-v2 is a small, pretrained causal language model with 1,405,008 parameters. It uses the LlamaForCausalLM architecture with six transformer layers, a hidden size of 144, and a vocabulary of 1,536 tokens.
The model is a base checkpoint and has not been instruction-tuned. Its maximum context length is 1,024 tokens.
Evaluation results
The following results were measured with zero-shot evaluation. The general benchmarks used lm-eval 0.4.12. ArithMark-3 was evaluated separately using its benchmark script.
| Benchmark | Metric | Score |
|---|---|---|
| ARC-Easy | acc_norm | 29.80% |
| ARC-Challenge | acc_norm | 22.78% |
| HellaSwag | acc_norm | 27.42% |
| PIQA | acc_norm | 53.54% |
| BoolQ | acc | 40.64% |
| SciQ | acc_norm | 53.30% |
| ArithMark-3 | acc_norm | 31.80% |
Evaluation was performed in float32 with a maximum context length of 1,024 tokens.
Training datasets
The main pretraining mixture contains approximately 6 billion tokens.
| Source | Tokens | Share | Role |
|---|---|---|---|
fineweb-edu (sample-10BT) | 2.70B | 45% | Educational web text |
Ultra-FineWeb-L3-en-Multi-Style-Synthetic | 1.80B | 30% | Synthetic multi-style text |
cosmopedia-v2 | 1.50B | 25% | Synthetic educational text |
An additional approximately 30 million tokens from lugman/add-sub-pre-training were used for arithmetic-focused pretraining.