Lumen-v2-130M-Base 💡

Lumen-v2-130M-Base is a high-efficiency Causal Language Model pretrained completely from scratch on a curated 6 Billion token multi-source educational corpus using Google TPU v5e-8.

The model features an 11-layer architecture with tied input/output embeddings, 4 full-rank Value Embedding (ResFormer) memory tables, causal depthwise convolution mixing (Canon K=3), and a single-digit tokenized vocabulary.

📊 Benchmark Results

All evaluations are zero-shot, evaluated with standard length-normalized likelihood (acc_norm) or raw accuracy over the full official test sets. Prompts are evaluated with the prefix <|bos|> token matching the pretraining distribution.

Core Benchmarks

Extended Task Breakdown

🍌 BananaMind Base Bench 1.1

Zero-shot evaluation on the BananaMind Base Bench 1.1 — a 350-example text-completion benchmark for base causal language models. Evaluation uses mean conditional log-probability scoring with no BOS token (official benchmark standard).

Category Breakdown

📌 Pretraining Data & Decontamination

The model was pretrained on 5,999,951,872 tokens across a balanced educational blend:

Decontamination Protocol

Full-corpus 13-gram exact-match filtering. All six mix sources were scanned with an Aho-Corasick automaton of 5,288,966 normalized 13-word phrases built from the test and validation splits of every benchmark reported in this card plus GSM8K (HellaSwag, ARC-Easy/Challenge, PIQA, LAMBADA, WikiText-2, SciQ, COPA, BoolQ, SWAG, WinoGrande, TruthfulQA, OpenBookQA, RACE, CommonsenseQA, MMLU, GSM8K, ArithMark-3). 7,590 rows (0.096% of the corpus) were removed across all sources; a post-cut re-scan found zero residual 13-gram overlaps. The released weights were retrained from scratch on the cleaned mix (identical architecture, tokenizer, data order and schedule; the ArithMark-3 generator itself has 0 exact / 0 digit-skeleton / 0 n-gram collisions with the benchmark by construction, and the held-out ArithWord split scores 86.95% vs 69.30% on the benchmark — generalization, not template recall).

🏗️ Architecture & Specifications

💻 Quickstart (Transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kefir090/Lumen-v2-130M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# bfloat16 needs compute capability >= 8.0 (Ampere or newer: RTX 30xx+, A100/H100).
# T4/P100 report is_bf16_supported()==True but produce wrong results -- check cc instead.
if torch.cuda.is_available():
    major, _ = torch.cuda.get_device_capability()
    dtype = torch.bfloat16 if major >= 8 else torch.float32
else:
    dtype = torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=dtype,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The solar system consists of"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.85,
        repetition_penalty=1.15,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Transformers version: tested with transformers>=4.57, including the latest 5.x line. Older releases may fail to load the custom architecture correctly.

Note: This is a pretrained base foundation model. It performs autoregressive text completion and is not instruction-tuned.

📜 License

This model and its weights are released under the Apache 2.0 License.