🐟 Fish LM: 68M Dense Small Language Model (SFT)
Govind222/68m-base-guppy-sft is a high-efficiency 68.96M parameter Dense Small Language Model (SLM) designed for ultra-fast edge inference, local device hosting, and conversational dialog.
Built from scratch using transformer-toolkit, the model was pretrained on 4 Billion tokens of FineWeb and instruction fine-tuned on 57,000 curated dialog examples (arman-bd/guppylm-60k-generic).
- Live Interactive Demo: https://fish-lm.runlog.in/ (hosted entirely on an edge Android smartphone)
- Training Notebook: Fish_LM.ipynb on GitHub
- Live Runlog Telemetry: solar-aurora-298 on Runlog
⚡ Highlights & Key Metrics
- Pure Dense Transformer: 100% dense computation with zero MoE overhead, maximizing per-token compute density and caching efficiency on edge CPU/GPU.
- Modern Architecture: Grouped-Query Attention (GQA), SwiGLU feed-forward layers, RoPE rotary positional embeddings, and pre-layer RMSNorm.
- Superb Perplexity: Achieves 1.275 SFT validation perplexity (
val_loss: 0.2429) on dialog tasks. - Extreme Efficiency: Full pretrain + SFT executed for under ~$14 on a single NVIDIA L40S GPU in ~14 hours.
- Mobile-Optimized: Serves real-time tokens at low latency (~35ms TTFT) on mobile ARM big.LITTLE CPUs.
📊 Evaluation & Benchmarks
Zero-shot evaluations performed via lm-evaluation-harness demonstrate that Fish LM 68M remains highly competitive with models nearly 2× its size:
| Benchmark | Metric | Fish LM (68M Dense) | Pythia 70M | Pythia 160M | GPT-Neo 125M | OPT 125M |
|---|---|---|---|---|---|---|
| PIQA | Accuracy | 58.1% | 58.6% | 61.4% | 62.3% | 61.8% |
| Winogrande | Accuracy | 48.7% | 51.6% | 50.4% | 49.5% | 51.5% |
| ARC-Easy | Acc Norm | 31.4% | 34.2% | — | 38.9% | 39.8% |
| Overall Average | Mean Acc | 39.6% | 41.4% | 43.7% | 43.5% | 43.9% |
⚙️ Model Architecture
| Parameter | Value | Specification Notes |
|---|---|---|
| Total Parameters | 68,957,056 (~68.96M) | Exact non-embedding + embedding tensor weight sum |
| Architecture Type | Dense Decoder-Only SLM | Pure dense transformer (no MoE routing) |
Layers (n_layers) | 12 | Uniform transformer blocks |
Hidden Dim (dim) | 512 | Model dimension ($d_{\text{model}}$) |
Query Heads (n_heads) | 8 | Head dimension = 64 |
KV Heads (n_kv_heads) | 2 | 4:1 GQA ratio for compressed KV cache memory |
| FFN Intermediate Dim | 1536 | $3 \times d_{\text{model}}$ |
| FFN Activation | swiglu | Dense SwiGLU gating (w1, w2, w3) |
| Positional Encoding | rope | Rotary Position Embeddings |
| Normalization | rmsnorm | Pre-norm with $\epsilon = 1\text{e-}6$ |
| Vocabulary Size | 32000 | Custom Rust BPE Tokenizer |
| Max Sequence Length | 512 | Context window length |
| Weight Tying | false | Untied input embeddings and output LM head |
📈 Training Details
1. Pretraining
- Base Checkpoint:
Govind222/68m-base - Pretraining Corpus: 4.0 Billion tokens from Hugging Face's
FineWeb (sample-10BT) - Objective: Standard autoregressive next-token prediction with cosine decay schedule
2. Supervised Fine-Tuning (SFT)
- Dataset: 57,000 instruction & conversation pairs from
arman-bd/guppylm-60k-generic - Training Framework:
transformer-toolkit - Telemetry & Tracking: Live monitored on Runlog (Run:
solar-aurora-298) - Hardware: 1× NVIDIA L40S GPU
- Training Throughput: ~6,064 tokens/sec
- Final SFT Validation Loss:
0.2429 - Final SFT Perplexity:
1.275
🚀 Quickstart & Inference
Prerequisites
pip install torch transformer-toolkit
Loading and Generating Text
import torch
from transformer_toolkit import TransformerConfig, Transformer, RustBPETokenizer
# 1. Initialize Architecture
config = TransformerConfig(
vocab_size=32000,
dim=512,
n_layers=12,
n_heads=8,
n_kv_heads=2,
ffn="swiglu",
hidden_dim=1536,
norm="rmsnorm",
pos_enc="rope",
dropout=0.0,
tie_weights=False,
max_seq=512,
use_kv_cache=True,
)
model = Transformer(config)
# 2. Load Weights & Tokenizer
state_dict = torch.load("model.pt", map_location="cpu")
if isinstance(state_dict, dict) and "model" in state_dict:
state_dict = state_dict["model"]
model.load_state_dict(state_dict)
model.eval()
tokenizer = RustBPETokenizer()
tokenizer.load("tokenizer.json")
# 3. Format Input & Generate
prompt = "<|im_start|>user\nHello! Who are you?<|im_end|>\n<|im_start|>assistant\n"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)
output_ids = model.generate(
input_ids,
max_new=50,
temperature=0.4,
top_k=50,
)
response = tokenizer.decode(output_ids[0].tolist())
print(response)
📱 Edge & On-Device Deployment
Fish LM was specifically benchmarked and optimized for constrained edge hardware:
- Host Architecture: Android / Termux on ARMv8/ARMv9 big.LITTLE core topology
- Thread Scheduling: Peak throughput achieved on 2 Big CPU cores (
OMP_NUM_THREADS=2), preventing thermal throttling and Little-core barrier latency. - Memory Footprint: Less than ~280 MB RAM at runtime with automatic idle watchdog unloading.
📚 Acknowledgments & Citations
- Built and trained with transformer-toolkit by @Barbade22.
- Dialog dataset provided by GuppyLM by Arman (@arman-bd).
- Experiment tracking and live dashboard provided by Runlog.