CyberSLM-33M-Base
A 33.5M-parameter cybersecurity-focused small language model, pretrained from scratch on a ~745 MB curated corpus (cybersecurity, computer fundamentals, programming, general English) with a custom 32k SentencePiece BPE tokenizer. Trained on a single NVIDIA L40S.
This is the base (pretrained-only) model. For the instruction-tuned
variant see cyberslm-33m-instruct.
Architecture
Decoder-only transformer (Llama-style, loadable with LlamaForCausalLM):
| Parameters | 33,531,264 |
| Hidden dim | 384 |
| Layers | 12 |
| Attention heads | 6 (MHA, head_dim 64) |
| FFN | SwiGLU, inner dim 1024 |
| Norm | RMSNorm (pre-norm), eps 1e-6 |
| Positional | RoPE, θ = 10,000 |
| Context | 4096 |
| Vocab | 32,000 (SentencePiece BPE, byte_fallback) |
| Weight tying | yes |
Training
- Corpus: ~665k documents (~205M tokens), 95/5 train/val split, seed 42
- 4,000 steps, effective batch 131,072 tokens/step (~524M tokens seen)
- AdamW β=(0.9, 0.95), wd 0.1, cosine schedule, peak LR 3e-4, grad clip 1.0
- Final val loss: 2.43
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("sabari2005/cyberslm-33m-base")
model = AutoModelForCausalLM.from_pretrained("sabari2005/cyberslm-33m-base")
ids = tok("A SQL injection attack is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))
Special tokens: pad=0 (<pad>), unk=1 (<unk>), bos=2 (<bos>), eos=3 (<eos>).
Limitations
Base model — no instruction tuning; it continues text rather than follows instructions. Trained on ~0.5B tokens, so expect limited general knowledge outside the cyber/CS domain. English only.