KL3M 500M Conservative, Step 100000
A 500M parameter language model trained with conservative hyperparameters using the Muon optimizer and spectral clamping. This checkpoint represents 100,000 training steps (8.32B tokens).
Model Details
- Architecture: Llama-based with Grouped Query Attention (GQA)
- Parameters: 500.3M
- Layers: 120
- Training Steps: 100,000
- Tokens Processed: 8.32B
- Precision: BF16
Training Configuration
Optimizer: Muon
- Muon LR: 0.000073 (depth-scaled)
- Aux LR: 0.00005
- Momentum: 0.95
Spectral Clamping
- Frequency: Every 100 steps
- Max condition: 2500
- Applied to: attention, MLP, and lm_head layers
Regularization
- Label Smoothing: 0.01
- Entropy Bonus: 0.005
- Activation Norm: 0.0005
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"alea-institute/kl3m-007-500m-step100000",
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("alea-institute/kl3m-007-500m-step100000")
inputs = tokenizer("The contract specifies", return_tensors="pt", return_token_type_ids=False)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.8, top_p=0.95)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
License
Apache 2.0