Vertex-0.6-35M-Base
A ≈34M-parameter base language model (Qwen3 architecture), pretrained from scratch on a single RTX 4060 Laptop GPU (8GB). Sibling of Vertex-0.6-100M, but fully independent: its own tokenizer, its own data pipeline, no shared weights or vocab.
Model
| Architecture | Qwen3 (Qwen3ForCausalLM) |
| Parameters | 33,924,224 (≈34M), tied embeddings |
| Hidden size | 384 |
| Layers | 10 |
| Attention heads | 6 (GQA, 2 KV heads, head_dim 64) |
| Intermediate size | 1536 |
| Context length | 1024 |
| RoPE theta | 10000 |
| Vocab | 32000, byte-level BPE |
Tokenizer
Independent byte-level BPE (tokenizer.json), vocab 32000, trained on a
≈480K-document sample of the pretraining mix (≈85% web / ≈15% code).
Special tokens: <|endoftext|> (bos/eos), <|pad|>. No chat/instruct
special tokens — this is a raw base model.
Training data
≈1.50B unique tokens, trained for 2 epochs (≈3.0B tokens seen, ≈88 tokens/param):
- Web (≈1.286B tokens, 85.7%): openbmb/Ultra-FineWeb-L1 (English, Apache-2.0)
- Code (≈214M tokens, 14.3%): codeparrot/codeparrot-clean (deduplicated Python)
Documents were concatenated with <|endoftext|> separators and packed into
fixed 1024-token blocks, shuffled independently each epoch.
Training recipe
- Single epoch-pair on one RTX 4060 Laptop 8GB, bf16 autocast + torch.compile
- AdamW (fused): lr 3e-3 peak, cosine decay to 10%, 300-step warmup, weight decay 0.1, betas (0.9, 0.95), grad clip 1.0
- Batch: 8 x 1024 tokens x 12 grad-accum = 98,304 tokens/step
- 30,517 optimizer steps total, ≈50.8K tok/s sustained (≈16h wall clock)
- Trained 2026-08-24 → 2026-08-25
Usage
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-0.6-35M-Base")
tok = PreTrainedTokenizerFast(
tokenizer_file=hf_hub_download("VertexResearch/Vertex-0.6-35M-Base", "tokenizer.json"),
bos_token="<|endoftext|>", eos_token="<|endoftext|>", pad_token="<|pad|>",
)
out = model.generate(tok("The", return_tensors="pt").input_ids, max_new_tokens=50)
print(tok.decode(out[0]))
Notes
- Base model only: no instruction tuning, no chat format, no safety tuning. For the instruction-tuned variant, see VertexResearch/Vertex-0.6-35M-Instruct.
- English + Python centric; other languages are effectively untrained.
Limitations
These models are not the most coherent yet and need more tuning: expect rambling, repetition, and inconsistent answers, especially over longer generations.