Vertex 0.6 100M — 4096-ctx Base
Base (text-completion) model of the Vertex 0.6 family by Vertex Research, pretrained from scratch on 10.01B tokens at 1024 context, then context-extended to 4096 via a two-stage rope ladder (2048: 350M tokens; 4096, rope theta 250K: 300M tokens) on fresh Ultra-FineWeb. This is the raw pretrain checkpoint: no instruction tuning, no chat alignment. Instruct and reasoning variants (with extended context) are released separately.
Model details
| Parameters | 96.75M (tied embeddings) |
| Architecture | Qwen3-based transformer |
| Hidden size / layers | 576 / 16 |
| Attention | 9 heads, 3 KV heads (GQA), head_dim 64 |
| FFN size | 2304 (SwiGLU) |
| Context length | 4096 |
| RoPE theta | 250,000 |
| Vocab | 32,768 (Supra2-100M tokenizer, digit-split) |
| Precision | trained fp32 compute-graph via torch.compile |
Training
- Data: ~10.01B tokens — English web text from Ultra-FineWeb plus a code slice, packed into 1024-token blocks, seeded shuffle, single epoch.
- Schedule: 101,819 optimizer steps, global batch 98,304 tokens (12 × grad-accum 8 × 1024), LR 3e-3 cosine → 3e-4, 300 warmup steps.
- Hardware: single RTX 4060 Ti 16GB, ~35K tokens/s, ~3.5 days wall clock.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("VertexResearch/Vertex-0.6-100M-4096-ctx-Base")
model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-0.6-100M-4096-ctx-Base")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=30)
print(tok.decode(out[0]))
This is a completion model — prompt it with text to continue, not with chat messages.
Limitations
These models are not the most coherent yet and need more tuning: expect rambling, repetition, and inconsistent answers, especially over longer generations.
97M parameters and 10B tokens: expect fluent English and basic factual/code completion, frequent factual errors, and repetition under greedy decoding. Knowledge cutoff of the web corpus is approximately April 2024.