BananaMind 2 Micro Preview - 100%
This is the 100% training preview of BananaMind 2 Micro, a
2,933,193-parameter base causal language model. It is not an
instruction-tuned chat model.
Architecture
| Field | Value |
|---|
| Parameters | 2,933,193 |
| Layers | 9 |
| Hidden size | 128 |
| Intermediate size | 512 |
| Query/KV heads | 4 / 2 |
| Head dimension | 32 |
| Context | 4,096 |
| Vocabulary | 2,048, tied |
| Attention | GQA, pre-RoPE QK norm |
| MLP | SwiGLU |
| Refresh | detached attention signal, kernel-9 causal depthwise gate, embedding reinjection |
The learned refresh scalar is initialized to zero. During cached generation,
the causal convolution history is carried alongside each layer's K/V cache.
Training
| Field | Value |
|---|
| Progress | 100% |
| Tokens seen | 74,998,349,824 |
| Target tokens | 75,000,000,000 |
| Hardware | 8 x NVIDIA RTX PRO 6000 Blackwell Server Edition |
| Matrix optimizer | Stock torch.optim.Muon |
| Muon LR | 0.05 |
| Embedding/1D optimizer | AdamW |
| AdamW LR | 0.004 |
| Precision | bfloat16 autocast |
| Token range | FineWeb-Edu | DCLM | Cosmopedia v2 | FinePhrase | FineMath | NPSet2 |
|---|
| 0.00B-15.00B | 55% | 30% | 6% | 4% | 4% | 1% |
| 15.00B-30.00B | 48% | 26% | 9% | 8% | 7% | 2% |
| 30.00B-48.75B | 42% | 20% | 12% | 12% | 11% | 3% |
| 48.75B-63.75B | 36% | 16% | 14% | 15% | 15% | 4% |
| 63.75B-75.00B | 30% | 12% | 15% | 18% | 21% | 4% |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind-Model-Previewers/BananaMind-2-Micro-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)