PicoLM-V3-82M-Instruct
Overview
PicoLM-V3-82M-Instruct is an ultra-compact, 82.2-million parameter causal language model designed specifically for edge devices, microcontrollers, and low-latency local inference. Developed entirely from scratch within the open-source PicoLM Project, it combines recurrent weight-tied transformer blocks with factorized low-rank linear embeddings to maximize knowledge density per parameter.
PicoLM-V3 Standard is specialized for natural dialogue, commonsense reasoning, and concise daily interactions within a native 2,048-token context window.
Model Specifications
| Attribute | Specification |
|---|---|
| Total Parameters | 82,233,792 unique parameters (tied I/O embeddings, 0 dead weights) |
| Physical Transformer Blocks | 21 layers |
| Effective Layers (Recurrent Pass) | 42 effective layers ($21 \times 2$ macro-loop passes) |
| Hidden Dimension ($d_{\text{model}}$) | 576 |
| Attention Architecture | Grouped-Query Attention (9 Query Heads, 3 KV Heads; GQA 3:1) |
| Head Dimension ($d_{\text{head}}$) | 64 |
| Feed-Forward Dimension ($d_{\text{ffn}}$) | 1,664 (SwiGLU activation) |
| Factorized Embedding Projection | $24,576 \rightarrow 128 \rightarrow 576$ (Rank-128 linear bottleneck) |
| Normalization | Pre-LN RMSNorm ($\epsilon = 10^{-5}$) with split-pass independent gains |
| Positional Encoding | Rotary Position Embeddings (RoPE, $\theta = 10,000.0$) |
| Context Length | 2,048 tokens native sequence length |
| Vocabulary Size | 24,576 BPE tokens |
Standardized Evaluation & Generational Progress
All models evaluated strictly using the EleutherAI LM-Evaluation-Harness standard:
| Model | Parameters | Tokens | Context | Legacy Scale (250 Q, Raw) | Official ARC-Easy (acc_norm) | PIQA (acc_norm) | HellaSwag (acc_norm) | ARC-Challenge (acc_norm) |
|---|---|---|---|---|---|---|---|---|
| PicoLM-V3 (Standard) | 82.2M | 758M | 2,048 | 44.80% | 39.32% | 58.49% | 36.00% | 23.63% |
| PicoLM-V3-Pro (Flagship) | 82.2M | 1.62B | 4,096 | 46.00% | 43.54% (Base) / 39.10% | 57.89% | 34.95% | 24.06% |
| PicoLM-V2.1-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | (legacy 250Q only) | ~56.5% | ~32.0% | ~22.5% |
| PicoLM-V2-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | (legacy 250Q only) | ~56.5% | ~32.0% | ~22.5% |
| GPT-2 (OpenAI) | 124M | ~10B | 1,024 | - | 31.40% | 62.80% | 31.50% | 22.10% |
| MobileLLM-125M (Meta) | 125M | 1.0T | 2,048 | - | 43.90% | 65.30% | 38.90% | 27.10% |
| SmolLM2-135M (HF) | 135M | 2.0T | 8,192 | - | 43.90% | 68.40% | 42.10% | 30.20% |
*PicoLM-V2 and V2.1 were evaluated exclusively on the legacy unnormalized 250-question sample. On the exact same 250-question scale, PicoLM-V3 outperforms V2 by +2.8 points (44.80% vs 42.00%).
Pre-Training & SFT Pipeline
1. Pre-Training (Part B - 758M Tokens)
- Data Mixture: Cosmopedia v2, FineWeb-Edu (score $\ge 4$), Python-Edu, and curated synthetic QA.
- Optimization: AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay 0.1), Cosine decay schedule to $1.0 \times 10^{-5}$, FP16 mixed precision.
2. Four-Pillar Supervised Fine-Tuning (SFT-Gold)
Aligned over 9,796 strictly filtered instruction conversations with assistant-only loss masking:
- Python Algorithmic Code (3,000 samples): Clean Python functions covering sorting, string manipulation, and list algorithms.
- Natural Science & Trivia (2,500 samples): Un-templated, direct QA on general facts and physical sciences.
- Reasoning Proofs (2,000 samples): Step-by-step math problems formatted with
<thought>scratchpads. - Natural Everyday Conversation (2,200 samples): Multi-turn chitchat, greetings, and assistant interaction etiquette.
- Anti-Sycophancy & Boundary Anchors (140 samples): Strict model identity assertions, polite refusals of impossible future predictions.
Quickstart & Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "aethertp/PicoLM-V3-82M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.float32,
device_map="auto"
)
messages = [
{"role": "user", "content": "There are 5 baskets. Each basket contains 6 pears. If 4 pears rot, how many good pears are left?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=80,
do_sample=False,
use_cache=False,
eos_token_id=2
)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response.strip())
Limitations & Ethical Considerations
- Scale Constraints: At 82.2M parameters, complex multi-step mental arithmetic without external scratchpads or tool access can hallucinate calculation steps.
- Language Coverage: The model is optimized strictly for English text and code.
- Decontamination: Training datasets were not decontaminated against benchmark test sets.
- No Native Tool Use: Cannot browse the web or execute live code.
Citation
@misc{picolm2026v3,
author = {Emre Polat and PicoLM Project Contributors},
title = {PicoLM-V3: Efficient Sub-100M On-Device Language Modeling},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/aethertp/PicoLM-V3-82M-Instruct}}
}