Particle 2.0
Particle 2.0 is a compact (~100M) chat model trained from scratch. It uses the same architecture as Particle 1.0.
This release is a further train plus a supervised fine-tune on a new dataset mix. Training data is not published.
Weights are released under MIT.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "prathamkode/particle-2.0"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
messages = [{"role": "user", "content": "hello"}]
prompt = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=False))
Model
| Architecture | Llama-style decoder (RoPE, SwiGLU, RMSNorm) |
| Parameters | 109.5M |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context | 2048 tokens |
| Tokenizer | Custom byte-level BPE, 32k vocabulary |
| Precision | bfloat16 |
| License | MIT |
The model is trained from random initialization. It is not a fine-tune of Llama, SmolLM, or any other public checkpoint.
Training
Continued training and a supervised fine-tune on a new dataset mix. The mix is not published.
Intended use
Research, evaluation, and small demos. Suitable for studying from-scratch training at ~100M scale.
Not intended as a production assistant, a source of facts, or a coding model.
Limitations
- Small capacity: weak on reasoning, long context, and tools
- Can hallucinate or contradict itself
- English-centric
- No preference tuning
Citation
If you use these weights, please cite Particle.