Vertex 0.6 200M — Base (Training Preview)
An in-progress pretraining checkpoint of Vertex 0.6 200M, the largest model in the Vertex 0.6 family so far. This is a raw base model mid-way through its pretraining run — published as a preview so anyone can watch the model take shape. It is not the final base and has had no instruction tuning.
Progress: step 202,000 of 305,175 (~66% of the pretraining run).
Model details
| Parameters | 198.21M (tied embeddings) |
| Architecture | Qwen3-based transformer |
| Hidden size / layers | 768 / 20 |
| Attention | 12 heads, 4 KV heads (GQA), head_dim 64 |
| Context length | 1024 |
| Vocab | 32768 |
| Precision | bf16 training, fp32 export |
Usage
Raw completion only — this is a base model, not a chat model.
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "VertexResearch/Vertex-0.6-200M-Base-Preview"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True,
temperature=0.6, top_p=0.9, repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))
Limitations
A mid-training checkpoint: fluent English with reasonable structure, but facts degrade quickly beyond common knowledge, it has not seen dedicated code training, and outputs can drift or confabulate freely. No instruction following, no chat format, no safety tuning. Expect the final base and its instruct variants to improve substantially on this checkpoint.