Vertex 0.6 5M Base

Base (pretrain-only) checkpoint of Vertex 0.6 5M, the smallest model in the Vertex 0.6 family at 4.77M parameters. Llama architecture: hidden 192, 9 layers, 3 heads (head_dim 64, no GQA), SwiGLU ffn 512, 4096-vocab ByteLevel BPE, tied embeddings, RoPE, ctx 512.

Pretrained from scratch on 500M tokens of English web text: Ultra-FineWeb, with Ultra-FineWeb-L3 synthetic rewrites (Multi-Style + QA) as a short final annealing phase, ~105 tokens per parameter, in 54 minutes on a single Apple M5 Pro (24 GB unified memory) using MLX at ~150k tokens/s.

This is a raw language model, no chat template, no instruction tuning. It is a research toy: fluent English at the sentence level, no factual knowledge. For something usable, see the larger family members such as Vertex-0.6-15M-Base and Vertex-0.6-100M-8192-ctx-Base.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "VertexResearch/Vertex-0.6-5M-Base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("The best way to learn programming is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Output from the run above:

The best way to learn programming is to deliver a comprehensive resource that enables readers to write effectively, effectively, and effectively. This resource provides a comprehensive understanding of programming methods, including advanced computing and automated

The model is 9.5 MB in bf16 and runs comfortably on CPU. Because the family is Llama-layout, it also converts to GGUF with llama.cpp's convert_hf_to_gguf.py.

Model details

The embedding matrix is kept small on purpose (16% of parameters) so most of the capacity is in the transformer blocks.

Tokenizer

tokenizer.json in Hugging Face tokenizers format, 4096 tokens, trained on a sample of the pretraining corpus. Token id 0 is <|endoftext|>: document separator in pretraining, stop token at inference.

Training data

500M tokens total, each seen once.

Training

Evaluation

Cross-entropy on held-out text with the model's own 4096-token vocabulary:

Held-out and training loss match, so nothing is memorized. Lower L3 numbers mean synthetic text is more predictable, not that the model is stronger there.

Sample outputs

Temperature 0.8, prompt in bold:

The best way to learn programming is through hands-on activities. The most effective approach involves seeing both TTHTO and TTHTO as a practical resource.

The company announced today that it had been obstructed by the presidential election of the military's Military Service. This marks a significant advancement in the construction of the state.

Water boils at a constant volumeter and porch table, and a minimum of two times each.

In 1969, astronauts were deactive Barnes and X, and shoes of the Gogri attended the joke, symbolizing the passage of time and celebration.

What it can and cannot do

Does: grammatical English with correct punctuation, matches the prompt's register (formal letter, press release, how-to), stays on topic for about one sentence.

Does not: recall facts (dates, names, numbers are invented), stay coherent past 15 to 20 tokens, follow instructions, write code.

Intended for education and research: tiny-model scaling, tokenizer effects, embedding-to-body ratios, or as a smoke-test model for inference engines. Not for any production or user-facing use.

Files

License

Apache 2.0.