Tyrian 75M

A 75M parameter decoder-only language model built entirely from scratch in PyTorch — no HuggingFace model classes, no nanoGPT wrapping. Every component (tokenizer, architecture, data pipeline, training loop, SFT) was written from scratch with Claude (Anthropic's AI assistant).

Model Details

Training

Pretraining

• ~17.8B tokens of English web text
• Data mix: FineWeb-Edu, Cosmopedia, StackExchange, Wikipedia, OpenWebText, WildChat, UltraChat, OASST2
• Custom BPE tokenizer (32K vocab, ChatML format)
• Cosine LR schedule: 3e-4 → 3e-5 with 2000-step warmup
• AdamW (β₁=0.9, β₂=0.95), weight decay 0.1
• Batch: 512K tokens/step
• Hardware: 2× RTX 5060 Ti 16GB, DDP

SFT Fine-tuning

• 100K examples from OpenHermes-2.5
• ChatML format with loss masking on user/system tokens
• LR: 2e-5 → 2e-6, 3 epochs

Benchmarks

Evaluated with log-likelihood scoring (no few-shot):

Comparable to GPT-2 (117M) at 0.64× the parameter count.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "redptam/tyrian-75m",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).cuda()

tokenizer = AutoTokenizer.from_pretrained("redptam/tyrian-75m", trust_remote_code=True)

# Chat (ChatML format)
prompt = "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=100, temperature=0.8, top_k=50,
                        stop_token_ids=(tokenizer.convert_tokens_to_ids("<|im_end|>"),))
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

Serving with tyrian-serve

tyrian-serve is an OpenAI-compatible server for the Tyrian models (/v1/chat/completions, with streaming), so it can be used from Open WebUI or any OpenAI client. It runs on CUDA and falls back to CPU.

git clone https://github.com/redptam/tyrian-serve && cd tyrian-serve
python3 -m venv .venv
./.venv/bin/pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu130
./.venv/bin/hf download redptam/tyrian-75m --local-dir .modelcache/tyrian-75m
MODEL_PATH=.modelcache/tyrian-75m MODEL_NAME=tyrian-75m ./.venv/bin/python -m uvicorn server:app --host 0.0.0.0 --port 8000

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"tyrian-75m","messages":[{"role":"user","content":"What is the capital of France?"}],"repeat_penalty":1.2}'

Set a repeat_penalty of about 1.2–1.3 in requests (or in the client's model settings); without one the 75M loops badly. The context window is 2048 tokens.

Special Tokens

Limitations

This is a small research model, built to learn how language models work from the ground up. It is not suitable for production use.

• Often wrong. It writes fluent text that is frequently factually incorrect, and it states errors confidently. Do not rely on it for medical, legal, financial or other advice.
• Not safety-tuned. It has had supervised fine-tuning only, with no preference or safety tuning, so it will not reliably decline harmful or inappropriate requests. Pretraining data includes web text and real chatbot conversations, so it can produce offensive, biased or otherwise inappropriate content.
• Repetition. Output can loop, especially with greedy decoding; sampling with a temperature helps.
• English only.
• Weak reasoning. Benchmark scores are close to chance on ARC-Challenge and WinoGrande (see above), and the context window is 2048 tokens.

License

MIT