[!IMPORTANT] This is an experimental research release, not a product. It is small, fluent but often factually wrong, and its image/speech capabilities are weak or unusable — see Results below before using it for anything beyond experimentation.
Learn more: a full write-up of how it works is on the Q-Project blog, and you can run it live in your browser — no server, no Python, no sign-in.
Model summary
Q-Omni-53M-Experimental is a 53.2M-parameter decoder-only model developed by Q-Project. Its weights are ternary ({-1, 0, +1}, quantisation-aware from step one of training), and instead of a trained embedding table it uses a frozen 512-bit fingerprint per token, shared across one 49,280-token vocabulary that spans text, image tokens and audio tokens. A fixed circuit resolves arithmetic, dates and similar exact operations inside the generated token stream. The architecture is our own; it was pretrained from scratch on a single V100 GPU. General small-model ternary-quantisation research (in the spirit of BitNet b1.58) inspired the weight format.
Results
| capability | result | verdict |
|---|---|---|
| exact calculation circuits (arithmetic, %, dates, weekdays, units, sort, compare, count) | correct <CALC> span 99.7%, exact final answer 98.7% (n=300) | works |
| fluent English text | FineWeb-Edu holdout perplexity 49.1 | fluent, not factual |
| ARC-Easy / PIQA (zero-shot) | 38.0% / 58.7% (chance 25% / 50%) | above chance |
| HellaSwag / ARC-Challenge / OpenBookQA | 27.5% / 22.7% / 25.0% (chance 25%) | at chance |
| image tokens (next-token loss) | 7.37 nats vs 9.01 for guessing | learns statistics; generated images are flat colour fields |
| audio tokens (next-token loss) | 3.98 nats vs 7.62 for guessing | learns statistics; speech synthesis not evaluated by ear |
| speech recognition | word error rate 123% | not usable |
Architecture
- 7 layers, hidden 896, 14 query / 2 KV heads (head dim 64), plain 2-matrix SiLU MLP 896→2464→896, per-head QK-norm, per-head sigmoid output gate on attention, RoPE (θ=1e5), context 4096 (trained at 2048).
- Ternary weights from step 1: every attention/MLP projection is re-quantised to {-1,0,+1}·α (per-row absmean α) at each forward pass with a straight-through estimator.
- Frozen fingerprint vocabulary: each token is a fixed ±1 code of 512 bits instead of a trained embedding. Input =
C[id]·W_in; readout =(h·W_out)·Cᵀ/√512 + frozen unigram log-prior. Ids ≥ 32768 (control, image, audio) carry an extra trainable correction on top of the frozen code. - Exact circuits (
circuits.py): the model writes<CALC>expr<EQ>;CircuitLogitsProcessorforces the exact result and<ECALC>— arithmetic, percentages, dates, weekday, unit conversion, comparison, sorting, letter/word counts.
Token layout
| ids | content |
|---|---|
| 0–32767 | text BPE (32k); 0 <pad>, 1 <bos>, 2 <eos> |
| 32768–32799 | control: <img> </img> <aud> </aud> <t2i> <i2t> <asr> <tts> <sep> <USER> <MODEL> <EOT> <CALC> <EQ> <ECALC> … |
| 32800–40991 | image codes (VQGAN, 16×16 = 256 tokens per image) |
| 40992–49183 | audio codes (4 codebooks × 2048, frame-interleaved, 12.5 frames/s) |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/Q-Omni-53M-Experimental") # adds <bos> automatically
model = AutoModelForCausalLM.from_pretrained("q-project/Q-Omni-53M-Experimental", trust_remote_code=True)
ids = tok("Once upon a time,", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True, top_k=40,
suppress_tokens=list(range(32768, model.config.vocab_size)))[0], skip_special_tokens=True))
For exact circuits, build [<bos>, <USER>, *question_ids, <EOT>, <MODEL>] and generate with circuits.CircuitLogitsProcessor(tokenizer).
Packed single-file weights (model.qpack, 33 MB)
The same model also ships as one compact file: ternary weights at 1.6 bit/weight plus the fingerprint table, tokenizer, config and a self-test. This repo (folder with model.qpack and no other weights) loads the same way through AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True); explicitly: QOmniForCausalLM.from_qpack("model.qpack"). A torch-free numpy runtime is included (qpack_run.py): python qpack_run.py model.qpack "Once upon a time,", or --selftest to verify it reproduces the reference model exactly.
Limitations
- Facts are frequently wrong; thin general knowledge; repetitive under greedy decoding; weak creative writing. No safety tuning.
- Image generation does not produce recognisable pictures; speech recognition is unusable; speech synthesis was checked only numerically, not by listening.
- Ternary weights at this size are a research setting: an fp16 model of the same size was not compared for this checkpoint.
- Trained on one GPU with limited compute; do not expect competitive benchmark scores.
License and data terms
Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check before commercial use: FineWeb-Edu and the SmolLM corpus (ODC-By 1.0 / Apache-2.0), CC3M (Google Conceptual Captions, research use), LibriSpeech (CC-BY 4.0), the Mimi audio codec (CC-BY 4.0) and the VQGAN image tokenizer (OpenRAIL++, use-based restrictions apply to that component).