Supra2-Medium-Base

Supra2-Medium Base is a 25M-parameter decoder-only language model pretrained from scratch by SupraLabs on 20B tokens of English web text. It uses the Qwen3 architecture with a custom 16,384-token tokenizer.

This is a base model. It has not been instruction-tuned, chat-tuned, or aligned in any way.

At only 25 million parameters, Supra2-Medium demonstrates that meaningful language modeling can be achieved with extreme parameter efficiency—trained at ~800 tokens per parameter, which is significantly higher than typical pretraining ratios. This makes it ideal for research into data-efficient scaling and ultra-lightweight deployments.

Let the model speak - a sample

Prompt: "Artificial intelligence (AI) is " Completion:

 Artificial intelligence (AI) is espoused by the AI community.
The AI community is a group of people who are interested in AI and are interested in the use of AI in the field of AI.
The goal of AI is to improve the quality of life of people in the field.
The aim of AI is the development of AI and the application of AI in a society.
The purpose of AI is that it can be used to improve the performance of the society.
It is a technology that is used to improve human intelligence.
The technology is used to make the human intelligence.

Evaluation & Benchmarks

All benchmarks were evaluated using the EleutherAI LM-Eval Harness.

Final Train Loss: 3.2469 (no eval loss available)

Note: The model shows strong performance relative to its size, particularly given the high token-per-parameter ratio. It even strongly competes with our previous 50M model!

Model Details

Architecture

Training Data

Documents were tokenized with the custom tokenizer_16k, concatenated into a flat uint16 token stream, and packed into contiguous 1,024-token chunks (no padding, no document masking — sequences may cross document boundaries).

Training Procedure

Key Design Decisions

• High token-per-parameter ratio: At 800 tokens/param, this model pushes the boundaries of data efficiency for small models
• Compact vocabulary: 16K vocab reduces embedding overhead while maintaining coverage
• Tied embeddings: Reduces parameter count and improves training stability
• Multi-head attention (MHA): Unlike larger Supra2 models using GQA, this architecture uses standard MHA for simplicity at this scale
• No sliding window: Full attention within the 1K context

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2-Medium"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.2,
        top_p=0.85,
        top_k=25,
        no_repeat_ngram_size=3,
        repetition_penalty=1.2,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

Tokenizer notes

The model uses a custom 16,384-token vocabulary optimized for English web text. The tokenizer is shared across the Supra2 family's smaller models for consistency and efficient multi-task fine-tuning.

Intended Use

Intended:

• Research on extreme parameter efficiency and data-efficient pretraining
• Ultra-lightweight edge deployments where memory is severely constrained
• Educational use for understanding transformer architectures at minimal scale
• Starting point for domain-specific fine-tuning when compute resources are limited
• Ablation studies on small model behavior and scaling laws

Not intended:

• Production deployment for critical applications
• Factual question answering or knowledge-intensive tasks
• Long-form coherent generation beyond a few paragraphs
• Non-English text (trained exclusively on English)
• Any application requiring safety guarantees or alignment

Limitations and Bias

• Very small. At 25M parameters, the model has extremely limited capacity. Expect frequent hallucinations, factual errors, repetitive outputs, and poor reasoning.
• Base model. No RLHF, no safety tuning, no refusal behavior. It will continue whatever text you give it, including harmful or offensive prompts.
• Web-derived data. FineWeb-Edu is a filtered CommonCrawl derivative and carries the biases, stereotypes, and factual errors of the open web.
• Short context. Trained exclusively at 1,024 tokens. Extrapolation beyond this length is untested and likely degraded.
• No document masking. Attention could cross document boundaries within a packed chunk, which slightly blurs document independence.
• English only. Performance on other languages will be poor to non-existent.
• High token-per-parameter ratio. While efficient, training at 800 tok/param means the model may be under-trained compared to models trained at lower ratios (e.g., 300 tok/param).

Performance Characteristics

Despite its tiny size, Supra2-Medium achieves surprisingly coherent short-form generation. Key observations:

• Strengths: Basic grammar, simple factoids, short completions, pattern matching
• Weaknesses: Complex reasoning, arithmetic, long-range coherence, factual accuracy, nuanced understanding
• Best use case: Generating 1-3 sentence completions, simple text transformations, educational demonstrations

Future work includes instruction-tuned variants and exploration of even more efficient architectures at this scale.

© SupraLabs 2026