ZeroS-Qana-5M

ZeroS-Qana-5M is a 4.94M-parameter base causal language model that studies content-dependent mixing across the expanded channels of a SwiGLU feed-forward block. It was trained as a compact generalist model and serves as a controlled architecture-research checkpoint. The final checkpoint was recorded at step 40,000 with a WikiText-103 normalized BPB of 1.4241.

Quick start

The model uses a custom Transformers architecture, so trust_remote_code=True is required.

pip install "torch>=2.5" "transformers>=4.50" safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "fromziro/ZeroS-Qana-5M"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

inputs = tokenizer(
    "The process of photosynthesis",
    return_tensors="pt",
).to(device)

with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64)

print(tokenizer.decode(output[0], skip_special_tokens=True))

The checkpoint is stored in native bfloat16 Safetensors. The exported model does not implement a KV cache, so autoregressive generation recomputes the visible context at each step.

Architecture

The expanded SwiGLU representation is divided into channel chunks that act as queries, keys, and values for a small dense mixing graph. Each 24-channel chunk is split across three heads, and each head computes an 18x18 content-dependent mixing matrix.

The centered operator is V_mix = [I + s(A(x) - A0)]V. The mixed representation then passes through the block's single SwiGLU down projection. Shared diagonal Q/K scales keep the mixer lightweight, while the reference distribution provides an identity-preserving initialization.

What is different?

Token-to-token attention remains a standard causal grouped-query attention layer. The architectural change is inside SwiGLU: after expansion, each token's features are divided into small channel chunks and content-dependent attention mixes information between those chunks. The centered update V_mix = [I + s(A(x) - A0)]V preserves the ordinary SwiGLU representation at its reference state while allowing learned cross-channel interactions. The mixed representation then passes through the block's single down projection, contracting it back to model width before it rejoins the residual stream.

Tokenizer

The model uses the 4,096-token tokenizer from AxiomicLabs/GPT-S-5M at revision 275b9c3ca78736bf6aeb154c7e2d5f5764fe9035. The exported checkpoint preserves its vocabulary, tokenization pipeline, and native special tokens. Standalone prompts receive the tokenizer's native BOS prefix.

Training

The model was trained for 40,000 updates on 20.97B tokens, with an effective batch of 524,288 tokens per update and a sequence length of 1,024.

Training used native BF16 compute, AdamW for embeddings and scalar parameters, Muon for hidden matrices, a 1,000-update warmup, a flat learning-rate phase through update 30,000, and cosine decay to zero at update 40,000.

Evaluation

The following results characterize the released checkpoint across language modeling, commonsense completion, reading comprehension, and science question answering. Results are reported task by task without composite aggregation.

All tasks were evaluated zero-shot with lm-eval 0.4.12 at model revision fa2cd38b2888846e350bf533dc7bd21fbcfc344f. Model weights were BF16, likelihood softmax was FP32, the evaluation batch size was 8, a native BOS token was added, and the maximum context was 1,024 tokens. Inputs exceeding the model context, encountered in BoolQ, were left-truncated by the harness. Standard errors are reported directly from lm-eval; n/a means the task does not define that metric.

BLiMP was evaluated over all 67 official subtasks with automatic batch-size selection; the table reports its group accuracy. ArithMark-3 was evaluated separately using its official acc_norm metric. A standard error was not recorded for that ArithMark-3 run and is therefore left as n/a.

WikiText-103 validation produced loss 3.1497, perplexity 23.33, and normalized BPB 1.4241 over 359,037 scored tokens using 1,024-token windows with a 512-token stride.

Intended use and limitations

ZeroS-Qana-5M is intended for architecture research, representation analysis, educational experiments, and controlled comparisons. It is a compact base model rather than an instruction-tuned assistant. Its scale and context length limit factual reliability, generation quality, and long-context capability. Outputs may contain inaccuracies or biases inherited from the training data and should not be used for consequential decisions.

Credits

ZeroS-Qana-5M was led, developed, trained, evaluated, and released by User01110. Additional authorship and project credit belong to Paul Courneya and Jonathon LY. The project is part of the FromZiro research community.

Copyright (c) 2026 User01110
Copyright (c) 2026 FromZiro
Copyright (c) 2026 Paul Courneya
Copyright (c) 2026 Jonathon LY

License

The complete release is covered by the Qana Open Attribution License 1.0, including the weights, architecture, implementation, configuration, tokenizer artifacts, metadata, and documentation.

• No permission or fee is required for private testing, research, evaluation, use, modification, fine-tuning, implementation, or commercial use.
• Redistribution, derivative models, and public products or services are also allowed, but must preserve the license and provide visible credit to ZeroS-Qana-5M and its authors: User01110, Paul Courneya, and Jonathon LY with a repository link where practical.
• Users are solely responsible for their use, outputs, compliance, and any misuse. The authors and FromZiro provide no warranty, accept no liability, and do not endorse downstream applications.

Third-party components, datasets and tokenizer materials remain subject to their respective licenses.

Citation

@misc{zeros_qana_5m_2026,
  title        = {ZeroS-Qana-5M},
  author       = {User01110},
  organization = {FromZiro},
  year         = {2026},
  url          = {https://huggingface.co/fromziro/ZeroS-Qana-5M}
}