TextModel-v1
A 122.7M parameter language model trained from scratch. TextModel-v1 is a Llama-style decoder-only transformer, built to be a strong, honestly-evaluated entry on small language model benchmarks, in particular GCI-Bench.
Architecture
| Type | LlamaForCausalLM |
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 (no GQA) |
| Intermediate size | 2048 |
| Max position embeddings | 2048 |
| Vocab size | 49152 |
| Tied embeddings | yes |
| Parameters | 122,702,592 |
Training
TextModel-v1 was trained from random initialization, not fine-tuned from any existing checkpoint. Training data was 1.64 billion tokens: 85% FineWeb-Edu, mixed with 15% synthetically generated documents covering causal reasoning across 20 topic domains and six causal structure templates. The synthetic data was generated fresh for this project and never overlaps with any published benchmark's actual test items. Sequences were packed to 1024 tokens, trained for 50,000 steps at batch size 32 on a single A100.
This checkpoint is step 50,000 of that run, chosen after sweeping all saved checkpoints against the benchmarks below and picking the one with the strongest overall result.
Evaluation
All numbers below are from the reference evaluation scripts run directly against this checkpoint, not self-reported estimates.
| Benchmark | Score |
|---|---|
| GCI-Bench (GCI score, full 5000-item test set) | 55.04 |
| BLiMP (accuracy) | 80.31 |
| ARC-Easy (accuracy) | 49.71 |
| WikiText-2 (byte perplexity) | 2.628 |
GCI-Bench measures whether a model's attention and gradient dynamics prioritize causally relevant context over distractor text, using the reference harness published at Glint-Research/GCI_Bench. The synthetic training data mentioned above was built independently to the same format as GCI-Bench (topics and causal structure), but the actual 5000 published test items were never used for training, only for this evaluation.
Weights as an image
model.png plus model_png.json encode the same weights as model.safetensors, losslessly, as float16 values packed into an RGB image. This is a stylistic convention shared across other releases in this line. png_codec.py can encode or decode either direction; round-trip max absolute error against the safetensors weights is under 2e-3, consistent with float16 precision.
Files
model.safetensors,config.json,tokenizer.json,tokenizer_config.json,generation_config.json: standard Hugging Face format, loadable directly withAutoModelForCausalLM.from_pretrained.model.png,model_png.json,png_codec.py: the weights-as-image encoding described above.pretrain_gci.py: the training script used to produce this checkpoint.gci_synth.py: the synthetic data generator used for the causal-reasoning portion of training.
Intended use
TextModel-v1 is a base model, not instruction-tuned or chat-tuned. It is intended for text completion and as a research artifact for studying small language model behavior, not for production use requiring factual reliability.
License
MIT.