modern-llm V1 (base, pretrained)
A 353M-parameter decoder-only transformer, built and trained from scratch — no Hugging Face Trainer, raw PyTorch. This is the base pretrained model: strong English fluency, but not instruction-tuned. It completes text; it does not follow instructions or answer questions like a chatbot.
Use it on this HF Space: https://huggingface.co/spaces/JohnEnev/modern-llm-playground
📖 Full writeup: Building a 350M Transformer From Scratch 💻 Code: github.com/JohnEnev/modern-llm
Architecture
| Parameters | 353,502,208 |
| d_model | 1024 |
| Layers | 24 |
| Attention heads | 16 |
| KV heads | 16 (standard MHA, no GQA) |
| Position encoding | RoPE |
| Normalization | RMSNorm, Pre-LN |
| Feed-forward | SwiGLU (8/3 hidden multiplier) |
| Context length | 1024 |
| Vocab size | 50,304 (tiktoken GPT-2 BPE, padded) |
| Optimizer | AdamW |
| Tied embeddings | Yes |
Training
- Data: FineWeb-Edu, 10B tokens (~29 tokens/parameter)
- Hardware: single H100
- Final validation loss: 2.86 (perplexity ≈ 17.5)
How to load
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import sys
local_dir = snapshot_download("JohnEnev/modern-llm-v1-base")
sys.path.insert(0, local_dir)
from modeling.gpt import GPT, GPTConfig
# NOTE: GPTConfig() defaults are shaped for V2 — override explicitly for V1.
config = GPTConfig(
vocab_size=50304, d_model=1024, n_layers=24, n_heads=16, n_kv_heads=16,
max_seq_len=1024, use_flash=True, tie_weights=True,
use_qk_norm=False, use_diff_attn=False, use_mhc=False, use_xsa=False,
)
model = GPT(config)
state_dict = load_file(f"{local_dir}/model.safetensors")
model.load_state_dict(state_dict, strict=False) # lm_head re-tied below
model.lm_head.weight = model.token_embeddings.weight
model.eval()
How to sample from it
import torch, tiktoken
enc = tiktoken.get_encoding("gpt2")
input_ids = torch.tensor([enc.encode("The meaning of life is")])
output = model.generate(input_ids, max_new_tokens=50, temperature=0.8, top_k=50)
print(enc.decode(output[0].tolist()))
Related checkpoints
modern-llm-v1-sft— instruction-tuned via SFTmodern-llm-v1-grpo— further trained with GRPO
Limitations
Base model, English only, 1024-token context, not instruction-tuned. See the SFT/GRPO checkpoints above for a usable assistant.