Lara-350M (Pretrained)
A 350M parameter LLaMA-style language model pretrained from scratch on a custom corpus. This is the base pretrained checkpoint — it has not yet been instruction-tuned or fine-tuned with any persona.
Architecture
- Type: LLaMA (decoder-only transformer)
- Parameters: 439,403,520
- Layers: 24
- Hidden size: 1024
- Attention heads: 16
- KV heads: 16
- Vocab size: 64,000
- Max sequence length: 2048
- Activation: SiLU (SwiGLU)
- Normalization: RMSNorm
Tokenizer
Important: This model uses a custom tokenizer trained from scratch on the training corpus. It is NOT a standard LLaMA, GPT-2, or any other pre-existing tokenizer.
- Type: BPE (Byte-Pair Encoding)
- Vocab size: 64,000
- Special tokens: BOS (
<s>), EOS (</s>), UNK (<unk>) - Training: Trained alongside the model on the same corpus
Do NOT use a standard LLaMA tokenizer with this model — the token IDs will be completely wrong and produce garbled output.
Training
- Architecture: Custom LLaMA-350M (trained from scratch)
- Training: 100,000 iterations with AdamW optimizer
- Batch size: 32 effective (2 micro-batch x 8 grad accum x 2 GPUs)
- Learning rate: 3e-4
- Hardware: 2x NVIDIA RTX 3060 (12GB VRAM each)
- Corpus: Custom pretraining corpus
What This Is Not
This is a raw pretrained language model. It has:
- No instruction tuning
- No persona or personality
- No RLHF or DPO alignment
- No guardrails or safety training
It predicts the next token based on patterns learned during pretraining. To use it as an assistant, it would need further fine-tuning (SFT, RLHF, etc.).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("0ldev/Lara-350M")
tokenizer = AutoTokenizer.from_pretrained("0ldev/Lara-350M")
inputs = tokenizer("The meaning of life is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
License
Apache 2.0