Pebble-50M-Chat-beta
Pebble-50M-Chat-beta is the chat-tuned version of Pebble-50M-beta, an experimental 50M-parameter language model designed to test how a larger Pebble architecture performs with a 16,384-token vocabulary and 16,384-token context window.
The base model underperformed Pebble-25M and, on some evaluations, Pebble-10M. Pebble-50M-Chat-beta was subsequently fine-tuned to improve its ability to follow instructions and engage in conversational interactions.
Model Details
- Architecture: Hybrid Mamba2 / Transformer
- Block Pattern: 3 Mamba2 blocks : 1 Attention block (repeating)
- Parameters: ~49,334,448 (50M)
- Hidden Dimension: 768
- Layers: 8 (6 Mamba2, 2 Attention)
- Vocab Size: 16,384 (Custom Byte-Level BPE)
- Context Length: 16,384
- Pretraining Tokens: ~25,000,000,000 (~25 Billion)
- Fine-tuning Tokens: ~250,000,000 (~250 Million)
- Fine-tuning Dataset: Smol-SmolTalk
- Optimizer: Muon (for 2D hidden weights) + AdamW (for embeddings, norms, and scalars)
- Precision: fp32 master weights with bf16 autocast
Dataset Sources
The base model was trained on a 25B-token subset of the following datasets:
| Dataset | Token Allocation | Share |
|---|---|---|
| FineWeb-Edu | 7.50 billion | 30% |
| DCLM | 5.00 billion | 20% |
| Cosmopedia-v2 | 3.75 billion | 15% |
| FineMath-4+ | 3.75 billion | 15% |
| FinePhrase | 3.00 billion | 12% |
| NPset | 2.00 billion | 8% |
| Total | 25.00 billion | 100% |
Fine-Tuning
Pebble-50M-Chat-beta was fine-tuned on approximately 250M tokens from Smol-SmolTalk to improve conversational ability and instruction following.
Benchmarks
The original benchmark logs for the base model were lost, so exact evaluation results are unavailable.
The chat model is primarily intended for conversational use and should not be directly compared with the base model on benchmarks without considering the effects of fine-tuning.
Usage
Pebble-50M-Chat-beta does not require the mamba-ssm library and is intended to be usable with standard PyTorch-based inference implementations.
It may run on CUDA GPUs, AMD GPUs, Intel GPUs, and CPUs depending on the inference framework and available hardware acceleration.
Status
This is a beta/experimental model. It is primarily intended for research, experimentation, and conversational use.
License
Apache 2.0