1386.ai.rocm — Plasma 1.1 (500M)
A 500M-parameter transformer trained from scratch on a GMKtec Evo X2 mini PC (AMD Strix Halo APU, Radeon 8060S, 128 GB unified memory) with ROCm/PyTorch. No pretrained weights, ~3 weeks wall-clock at ~4,750 tok/s on the integrated GPU. Code: a ROCm port of eb1386/1386.ai.
What's in this repo
| File | Checkpoint | Notes |
|---|---|---|
plasma-1.1-ft-f16.gguf | instruction-tuned final (step 30k) | GGUF f16, llama.cpp-ready |
plasma-1.1-pretrain-f16.gguf | pretrain final (step 200k) | GGUF f16, llama.cpp-ready |
model.safetensors | instruction-tuned final | HF LlamaForCausalLM layout, f16, tied embeddings |
config.json | — | 1.1 architecture (26×1280, GQA 20/4, vocab 48k) |
tokenizer.model | — | SentencePiece, 48k, trained on the 1.1 corpus |
Architecture: 26 layers, hidden 1280, 20 query / 4 KV heads (GQA), head_dim 64, SwiGLU intermediate 3584, RMSNorm, RoPE θ=10000, context 1024, vocab 48000. ~575M parameters counting the tied embed/output matrix.
Training: pretrain 200k steps (FineWeb-Edu-hq, Wikipedia, code, books, arXiv shards), then instruction-tune 30k steps on synthetic User: …\nAssistant: … data with loss masked to assistant tokens.
Perplexity (llama.cpp, instruction-prose slice): ft 3.0, pretrain 13.4.
Honest limitations
- Format > facts. The ft model writes well-shaped markdown answers but is frequently wrong or hallucinated (it will "reverse a string" with a
try/except+ JSON example), and it likes to emit EOS early on instruct prompts. A 30k-step tune at 1.5e-5 LR mostly buys style. - 1024-token context. Chat format is plain text —
User: question\nAssistant:— no chat template, no special tokens beyond the SentencePiece defaults. - The pretrain GGUF is base-model continuation only; use the ft one for instruct-style prompts.
RoPE caveat (important for conversions)
The training code rotates RoPE pairs as (2i, 2i+1) (GPT-J/interleaved style), while llama.cpp's llama arch expects split-half (i, i+32) pairing. These GGUFs have q/k_proj rows permuted to compensate — so llama.cpp's default rotary settings work correctly, and --rope-* overrides are neither needed nor wanted. If you re-convert a checkpoint with a pre-fix convert.py, the file loads but emits control-token garbage (PPL ~3000). Repo commit 4c32c49 contains the fix; verify any conversion with llama-perplexity.
Usage
llama-cli -m plasma-1.1-ft-f16.gguf -p "User: What is the capital of France?\nAssistant:" -n 100
Works with any llama.cpp build ≥ b5000. f16 → q8_0 is ~600 MB, q4_k_m ~330 MB, both fine at this size.
Trained May–June 2026 on a Strix Halo, in a docker container, on integrated graphics, because someone had to.