Veyra 30M Instruct
Veyra-30M-Instruct is the first public instruction-tuned checkpoint in the Veyra 30M line.
It is built on top of veyra-ai/Veyra-30M-Base and is intended to test how far a 30M-parameter English base model can be pushed with masked ChatML supervised fine-tuning.
Model Details
- Base model:
veyra-ai/Veyra-30M-Base - Parameters: approximately 34.6M
- Language: English
- Context length: 1024 tokens
- Architecture: decoder-only causal language model
- Chat format: ChatML
- License: Apache 2.0
- Status: experimental instruct release
Architecture
Veyra-30M is a compact decoder-only transformer-style model.
- Vocabulary size: 8,192
- Hidden size: 512
- Layers: 8
- Layer pattern: alternating attention and MLP-only blocks
- Query heads: 8
- Key/value heads: 2
- Attention type: grouped-query attention
- MLP: SwiGLU
- Normalization: RMSNorm
- Positional encoding: RoPE
- Context length: 1024 tokens
- Tied input/output embeddings
- KV cache support for generation
Training
This checkpoint was trained with masked ChatML supervised fine-tuning.
- Base checkpoint:
veyra-ai/Veyra-30M-Base - SFT tokens: 100M non-padding SFT tokens
- Supervised assistant tokens: approximately 35M supervised assistant tokens
- Loss masking: system and user turns were used as context but masked from loss; assistant responses were supervised.
The released checkpoint was selected from intermediate SFT milestones based on qualitative behavior, not only lowest training loss. Later SFT checkpoints reached lower loss but showed more generic refusal/template behavior, so checkpoint selection was based on output quality.
Intended Use
- Tiny-model instruction-following experiments
- Local assistant experiments
- ChatML behavior testing
- Simple formatting and JSON experiments
- Basic Python/helpfulness prompts
- Studying SFT behavior in very small language models
Not Intended For
- Production applications
- Safety-critical use cases
- Medical, legal, financial, or security advice
- Reliable factual QA
- Long-context reasoning
- Robust arithmetic
- User-facing deployment without additional safeguards
Known Limitations
This is a 30M-parameter model and has significant limitations.
Known failure modes include:
- Weak variable binding
- Weak arithmetic
- Weak multi-turn recall
- Occasional repetition
- Confident but incorrect answers
- Generic refusal or disclaimer behavior
- Tool-call or reasoning-template contamination
- Sensitivity to prompt wording
- Fluent nonsense on unfamiliar prompts
This model is not fully safety-tuned. It may refuse some harmful requests, but refusal behavior is not reliable.
Prompt Format
This model uses ChatML.
<|im_start|>system
You are Veyra, a tiny local instruction model. Be concise, useful, casual, and lightly playful. Correct mistakes gently.<|im_end|>
<|im_start|>user
What does a tokenizer do?<|im_end|>
<|im_start|>assistant
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "veyra-ai/Veyra-30M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
messages = [
{
"role": "system",
"content": "You are Veyra, a tiny local instruction model. Be concise, useful, casual, and lightly playful."
},
{
"role": "user",
"content": "Explain what an API is using a simple analogy."
},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=120,
temperature=0.3,
top_k=40,
do_sample=True,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|im_end|>")],
use_cache=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
Special Tokens
Important tokenizer special tokens include:
<|bos|>
<|eos|>
<|pad|>
<|unk|>
<|im_start|>
<|im_end|>
<|tool_call|>
<|tool_result|>
<|context|>
<|reasoning|>
<|end_reasoning|>
<|answer|>
<|fim_prefix|>
<|fim_middle|>
<|fim_suffix|>
For this checkpoint, standard ChatML with <|im_start|> and <|im_end|> is the recommended format.
Citation / Attribution
If you use or build on this model, please retain attribution to Veyra AI.
License
Apache 2.0.