Supra2-Nano

Supra2-Nano is an 800K-parameter language model trained from scratch on 1B tokens using the Qwen3 architecture. It is part of the Supra2 family, a set of small language models trained on consumer-grade hardware to study scaling behavior at extremely low parameter counts.

Model summary

The model was trained with a data mix dominated by FineWeb-Edu, supplemented with a small proportion of Cosmopedia-v2 for synthetic instructional-style text.

Architecture

Supra2-Nano uses the Qwen3 architecture at a heavily downscaled size: reduced hidden dimension, few attention heads, and a small number of transformer layers, all fit to an 800K parameter budget. Grouped-query attention and RMSNorm are used as in the original Qwen3 design. The tokenizer vocabulary is kept small (4,096 tokens) to avoid the embedding and output projection layers dominating the parameter count at this scale, a common failure mode when applying full-size tokenizers to tiny models.

Training

• Dataset: 99% FineWeb-Edu, 1% Cosmopedia-v2
• Tokens seen: 1B (single epoch, no repetition)
• Steps: 7,000
• Hardware: 2x NVIDIA T4 GPUs (Kaggle)
• Objective: standard next-token prediction (causal LM)

At this parameter count, the model is not expected to produce coherent long-form generations. The purpose of this training run is to establish a baseline for the Supra2 family and to validate the training pipeline (tokenizer, architecture, data mix) before scaling to larger variants in the same family.

Benchmarks

Zero-shot evaluation on standard small-LM benchmarks, compared against two other sub-1M/1M-parameter models from the same lab:

Scores across all three models sit close to random/majority-class baselines, which is expected at this parameter scale. PIQA is the strongest signal, consistent with what's typically observed in sub-million-parameter models — it has the lowest reasoning depth requirement of the four tasks. Supra2-Nano performs comparably to Supra-Mini-v6 despite having 20% fewer parameters, and outperforms Supra-Mini-v3 on three of four tasks.

Get started

import json
import os

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, LlamaTokenizerFast

MODEL_NAME = "SupraLabs/Supra2-Nano"
PROMPT = "The history of artificial intelligence begins"


def load_tokenizer(model_name: str) -> LlamaTokenizerFast:
    local_dir = snapshot_download(model_name)
    config_path = os.path.join(local_dir, "tokenizer_config.json")

    with open(config_path, "r", encoding="utf-8") as f:
        cfg = json.load(f)

    if isinstance(cfg.get("extra_special_tokens"), list):
        cfg["extra_special_tokens"] = {}
        with open(config_path, "w", encoding="utf-8") as f:
            json.dump(cfg, f, indent=2, ensure_ascii=False)

    return LlamaTokenizerFast.from_pretrained(local_dir)


def main():
    device = "cuda" if torch.cuda.is_available() else "cpu"

    tokenizer = load_tokenizer(MODEL_NAME)
    model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.bfloat16)
    model.eval().to(device)

    inputs = tokenizer(PROMPT, return_tensors="pt").to(device)
    inputs.pop("token_type_ids", None)

    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=100,
            do_sample=True,
            temperature=0.8,
            top_p=0.9,
            repetition_penalty=1.3,
            pad_token_id=tokenizer.eos_token_id,
        )

    print(tokenizer.decode(output_ids[0], skip_special_tokens=True))


if __name__ == "__main__":
    main()

Intended use

This model is intended for research into small-model scaling, architecture ablations, and training pipeline validation. It is not intended for production use or any application requiring reliable text generation. Outputs at this scale will frequently be repetitive, ungrammatical, or incoherent.

Limitations

• 800K parameters is well below the threshold where transformer language models produce fluent text.
• The 4,096-token vocabulary increases out-of-vocabulary fragmentation on rare words and non-English text.
• Single-epoch training on 1B tokens with no held-out validation set beyond benchmark evaluation.
• No instruction tuning or alignment; this is a base (pretrained) model only.

Citation

If you use this model, please cite the SupraLabs organization on Hugging Face.