PURCAR Thanatos 0.1
Thanatos 0.1 is a custom causal Transformer trained by the PURCAR project.
- 202,564,432 trainable parameters
- 48 Transformer encoder layers used causally
- hidden size 512
- 8 attention heads
- feed-forward size 2048
- ByteLevel BPE vocabulary of 50,000 tokens
- context window of 1,024 tokens
The original checkpoint was jelli_best_1.pt. Optimizer and scheduler state
were intentionally excluded from model.safetensors.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "ihatebaselines/purcar-thanatos-0.1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
model.attach_tokenizer(tokenizer)
reply = model.generate(
"User: What are you?\nAssistant:",
temperature=0.67,
max_new_tokens=80,
)
print(reply)