Veyra 30M Base

It is not instruction tuned and should not be evaluated like a finished chat assistant. It is expected to hallucinate, repeat, fail simple factual/math prompts, and continue text in odd ways.

Training summary

Approximate training stages:

• 1B tokens: Cosmopedia v2 bootstrap pretraining.
• +1.5B tokens: mixed continuation using Cosmopedia-v2 repository configs including cosmopedia-v2, fineweb-edu-dedup, and python-edu.
• +2.5B tokens: Went back to Cosmopedia v2 but increased context length from 512 -> 1024.
• Total: about 5B pretraining tokens.

Architecture

Veyra-30M is a small attention-sparse decoder-only language model.

Key details:

• Parameters: 34.6M
• Vocabulary: 8,192 tokens
• Hidden size: 512
• Layers: 8
• Attention heads: 8 query heads, 2 KV heads
• MLP intermediate size: 2048
• Activation: SwiGLU
• Normalization: RMSNorm
• Position encoding: RoPE
• Tied token embeddings / LM head
• Context in this checkpoint: 1024 tokens

Loading

This repository uses custom Transformers code.

Minimal usage:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo = "veyra-ai/Veyra-30M-Base"

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32)
model.eval()

prompt = "Photosynthesis is the process by which"
input_ids = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")

with torch.no_grad():
    out = model.generate(
        input_ids,
        do_sample=True,
        temperature=0.5,
        top_k=30,
        repetition_penalty=1.15,
        no_repeat_ngram_size=2,
        max_new_tokens=80,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

For raw completion prompts, use add_special_tokens=False.

Optimizer

Training used:

• CosineGatedAdam / CGA-v0 on 2D projection matrices
• AdamW on embeddings, norms, tied head, and auxiliary parameters

Intended use

This checkpoint is primarily for:

• continued pretraining
• research / ablations
• tracking Veyra training milestones
• testing tiny model behavior

It is not intended for production use or reliable factual answering.

Known limitations

This model can:

• hallucinate confidently
• repeat phrases
• fail arithmetic
• fail simple factual questions
• produce fake code
• continue in textbook-like or tutorial-like styles

Special Tokens

Important tokenizer special tokens include:

<|bos|>
<|eos|>
<|pad|>
<|unk|>
<|im_start|>
<|im_end|>
<|tool_call|>
<|tool_result|>
<|context|>
<|reasoning|>
<|end_reasoning|>
<|answer|>
<|fim_prefix|>
<|fim_middle|>
<|fim_suffix|>

Citation / Attribution

If you use or build on this model, please retain attribution to Veyra AI.

License

Apache 2.0.