ObsidianSmall-Base — Multiscreen 8.7M
ObsidianSmall-Base is an 8.7M-parameter English base language model pretrained from scratch on 3,999,989,760 tokens.
It is the first public checkpoint in this model family built entirely from Multiscreen blocks.
This is a base completion model, not an instruction-tuned or chat-aligned assistant.
Model details
| Property | Value |
|---|---|
| Architecture | Pure Multiscreen |
| Parameters | 8,716,506 |
| Blocks | 9 |
| Hidden width | 256 |
| Query heads | 8 |
| Head size | 32 |
| Key / value dimensions | 16 / 128 |
| Query chunk / window | 128 / 256 |
| MLP / FFN | None |
| Context length | 1,024 |
| Vocabulary size | 4,096 |
| Training tokens | 3,999,989,760 |
| Validation loss / PPL | 2.592 / 13.359 |
Architecture
This model implements and adapts the Multiscreen architecture introduced in Screening Is Enough by Ken M. Nakanishi.
Multiscreen replaces conventional self-attention and MLP/FFN sublayers with its own sequence-mixing blocks.
The implementation may differ from the architecture and training configuration described in the original paper.
Benchmarks
Zero-shot task results were measured with lm-evaluation-harness.
| Benchmark | Accuracy | Normalized |
|---|---|---|
| BLiMP | 71.891% | — |
| HellaSwag | 26.957% | 27.415% |
| ARC Easy | 33.712% | 33.291% |
| ARC Challenge | 17.833% | 21.843% |
| PIQA | 56.583% | 55.114% |
| SciQ | 70.700% | 60.000% |
| ArithMark-2.0 | 26.160% | — |
| ArithMark-3.0 | 28.800% | — |
WikiText-2 perplexity
Full WikiText-2 evaluation using rolling log-likelihood with a maximum context length of 1,024 tokens.
| Metric | Result |
|---|---|
| Word perplexity | 125.057 |
| Byte perplexity | 2.467 |
| Bits per byte | 1.303 |
Install
git lfs install
git clone https://huggingface.co/Dream-W/ObsidianSmall-Base
cd ObsidianSmall-Base
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
For NVIDIA CUDA with Triton:
python -m pip install -r requirements-gpu.txt
For evaluation:
python -m pip install -r requirements-eval.txt
The repository includes the required custom PyTorch/LitGPT runtime. It is not currently compatible with transformers.AutoModelForCausalLM.
Generate text
CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u generate.py \
--backend triton \
--device cuda \
--prompt "Once upon a time" \
--max-new-tokens 128 \
--temperature 0.8 \
--top-k 40
Interactive completion:
CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u generate.py \
--backend triton \
--device cuda \
--interactive
CPU execution is available with MULTISCREEN_BACKEND=torch, --backend torch, and --device cpu.
Evaluate
The bundled evaluator accepts any compatible lm-evaluation-harness task through --tasks.
MODEL="$PWD"
mkdir -p "$MODEL/results"
CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
TOKENIZERS_PARALLELISM=false \
PYTHONPATH="$MODEL/runtime:$MODEL/evaluation" \
python -u "$MODEL/evaluation/lm_eval_obsidian_official.py" \
--checkpoint-dir "$MODEL" \
--tokenizer-dir "$MODEL" \
--tasks hellaswag \
--batch-size 32 \
--device cuda \
--max-length 1024 \
--progress-every 100 \
--output "$MODEL/results/hellaswag.json"
Multiple tasks:
--tasks "hellaswag,arc_easy,arc_challenge,piqa,sciq,blimp"
Custom continuation-style multiple-choice datasets can be evaluated with evaluation/custom_mcq.py.
CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u evaluation/custom_mcq.py \
--model-dir . \
--hf-dataset AxiomicLabs/Arithmark-3.0 \
--split train \
--context-field ctx \
--choices-field endings \
--label-field label \
--group-by difficulty topic \
--backend triton \
--device cuda \
--output results/arithmark-3.json
Backends and decoding
| Backend | Setting | Use |
|---|---|---|
| Automatic | MULTISCREEN_BACKEND=auto | Select a compatible backend |
| Triton | MULTISCREEN_BACKEND=triton | Optimized NVIDIA CUDA execution |
| PyTorch | MULTISCREEN_BACKEND=torch | CPU and compatibility fallback |
Incremental Multiscreen KV-cache decoding is not implemented. Generation uses full-sequence decoding and processes the current context again for every generated token.
Training data
The pretraining curriculum included FineWeb-Edu, FinePDFs-Edu, DCLM, FineWiki, NVIDIA Common Crawl Code, and NVIDIA Common Crawl Math.
Limitations
The model has only 8.7M parameters and may produce repetitive, inaccurate, or incoherent text. It is English-focused, has no instruction-following or safety alignment, and was trained for a maximum context length of 1,024 tokens.