Qwen3-Omni-30B-A3B (Thinker) SAMI-AsymQ GGUF

📄 Technical Report (Preprint, Mathematical Proofs & Benchmarks):
Spiridon, V. (2026). SAMI Omni: Asymmetric Mixture-of-Experts Topology and Memory-Decoupled Multimodal Inference on Commodity Hardware. Technical Report, TSV Consulting SRL.
Permanent DOI: https://doi.org/10.5281/zenodo.23293547 · Zenodo Record: zenodo.org/records/23293547

This repository provides GGUF quantizations of the reasoning and visual backbone (Thinker) from Qwen/Qwen3-Omni-30B-A3B-Instruct, prepared by Valentin Spiridon (TSV Consulting SRL, Timișoara, Romania; ORCID: 0009-0000-2336-8527) using the SAMI (Structured Architecture for Modular Inference) methodology.

The checkpoint is configured to execute locally on consumer workstations equipped with a 6GB VRAM GPU (such as an NVIDIA GeForce GTX 1060 6GB) and 32GB system RAM using llama-server (or llama-cli) with CPU-MoE offloading (-cmoe).

Scope and Modality Specification

• Included Modalities (In GGUF): Text understanding/generation and Multimodal Vision (SigLIP projector).
• Audio Synthesis Notice: Native acoustic waveform synthesis from the upstream Qwen3-Omni Talker module (3.3B parameters + vocoder) is not bundled inside this GGUF file, as llama.cpp natively evaluates the LLM and Vision graphs. End-to-end voice interfaces require coupling this server with external TTS/STT pipelines.
• Nomenclature Clarification (30B vs. 35B): The upstream base repository designates the model family as 30B-A3B. A full census of all un-sharded tensors across all submodules (Thinker text backbone + visual projector + audio encoder + talker speech module) totals 35,259,818,545 parameters (approx. 35.26B). This GGUF specifically packages the 30.5B Thinker backbone (approx. 3.2B active parameters per forward pass).

Available Checkpoints and Hardware Allocation

Architectural Telemetry and Bandwidth Verification

A common question regarding running large models on consumer hardware is memory bandwidth feasibility. The physical math confirms why this architecture achieves approx. 16 tokens/second on DDR4 RAM:

• Total Parameters: 30.53 billion (Thinker backbone)
• Architecture: 48 Transformer layers, 128 sparse routed experts per layer ($48 \times 128 = 6,144$ total experts), intermediate dimension $d_{\text{ff}} = 768$.
• Active Routing: Top-8 active experts per token out of 128 (approx. 3.2B active parameters total per forward pass).
• Dense Footprint (VRAM): All self-attention projections, RMSNorm tensors, and Router Gate matrices (approx. 1.54B parameters) reside in GPU GDDR5 memory (192 GB/s bandwidth), requiring approx. 2.4 GB VRAM.
• Sparse Stream (DDR4 RAM): For each generated token, the CPU only streams the weights of the 8 activated experts (approx. 1.8B parameters).
• Data Transfer per Token: At asymmetric Q3_K / Q4_K precision, 1.8B parameters translate to approximately 1.4 GB to 1.6 GB of weights transferred per forward pass.
• Theoretical Throughput over Dual-Channel DDR4: $$\text{Throughput} = \frac{\text{Host Memory Bandwidth (30–35 GB/s)}}{\text{Active Expert Weights per Token (1.4–1.6 GB)}} \approx 16\text{–}18\text{ tokens/second}$$

Because inactive experts are never touched during a forward pass, memory bandwidth is not saturated by the full 14.8 GB file, enabling sustained 16 tokens/second execution on an Intel i5-8400 CPU paired with an NVIDIA GTX 1060 6GB.

Post-Training Weight Interventions

Prior to asymmetric quantization, the model weights underwent targeted directional tensor intervention across the 13 raw BF16 shards to adapt behavior for technical problem solving and eliminate refusal boilerplate:

1. Excised Pathways

• 6,259 Refusal Feature Dimensions (approx. 0.13% of expert columns): Identified via contrastive activation projection against refusal tokens across the 4,718,592 candidate expert columns ($6,144 \text{ experts} \times 768 \text{ intermediate dimensions}$). Projections with $z > 2.0$ along refusal trajectories were ablated in down-projection matrices ($W_{\text{down}}$), eliminating unsolicited boilerplate lecturing while preserving underlying capability.
• 10,128 Dormant Expert Dimensions (approx. 0.21%): Inactive parameter channels with near-zero activation variance ($\sigma^2 < 10^{-5}$) across standard calibration sets were set to zero to reduce residual stream noise.
• 1,491 Speculative Vectors: Attenuated to suppress token hallucination rates during domain-specific reasoning.

2. Enhanced Pathways

• 32,789 Technical Feature Dimensions (approx. 0.69%): Expert parameters exhibiting strong activation correlation with low-level systems programming (concurrency, memory management, pointers) and natural Romanian morphosyntax were directionally scaled.
• Manifold Stabilization: Following parameter ablation, output projections were aligned to prevent representation drift in the residual stream.

3. Latency Optimization: Bypassing Thought Tokens

• The base checkpoint generates an internal hidden reasoning trace (<thought> ... </thought>), which can take 20–40 seconds before outputting the first visible character on CPU/DDR4 systems.
• This release is calibrated to execute with --reasoning off --reasoning-budget 0, suppressing hidden monologue tokens and achieving a verified Time-To-First-Token (TTFT) of 2.08 seconds on commodity hardware.

Quantization Specification (AsymQ)

To preserve routing fidelity and prevent expert selection degradation, weights are quantized with layer-specific precision:

• Router Gate Matrices (ffn_gate_inp): Q8_0 (Guarantees numerical stability in expert routing decisions).
• Embeddings and LM Head (token_embd, output): Q8_0.
• Self-Attention Projections (attn_q, attn_k, attn_v, attn_output): Q5_K.
• Expert Gate and Up Projections (ffn_gate_exps, ffn_up_exps): Q3_K.
• Expert Down Projections (ffn_down_exps): Q4_K (Preserves residual stream projection accuracy).

Multimodal Vision Projector Alignment

Standard Qwen3-VL projectors concatenate multi-layer SigLIP features ($2048 \times 4 = 8192$), producing a dimension mismatch against a $d=2048$ text backbone.

• The SAMI visual projector isolates the primary merger pooler (thinker.visual.merger), aligning visual tokens directly to $d=2048$.
• Using --no-mmproj-offload, the 861 MB projector resides in host DDR4 RAM, enabling image analysis with 0 MB GPU VRAM overhead.

How to Run

Requires llama.cpp (build b11300 or newer) which includes support for -cmoe (--cpu-moe) and --reasoning-budget.

1. Launch llama-server

./llama-server \
  -m "Qwen3-Omni-35B-A3B-SAMI-AsymQ.gguf" \
  --mmproj "mmproj-SAMI-Omni-Vision-BF16.gguf" \
  --no-mmproj-offload \
  -ngl 99 \
  -cmoe \
  -t 4 \
  -c 4096 \
  --reasoning off \
  --reasoning-budget 0 \
  --host 127.0.0.1 \
  --port 8080

Parameter Explanation:

• -ngl 99 -cmoe: -ngl 99 offloads transformer layers to the GPU, but -cmoe (--cpu-moe) explicitly retains all 6,144 sparse expert matrices in host system RAM. Only dense attention projections, RMSNorm, and router gate heads are placed on the GPU (approx. 2.4 GB VRAM allocated out of 6GB), preventing CUDA Out-Of-Memory errors.
• --no-mmproj-offload: Keeps the 861 MB SigLIP vision projector in host DDR4 RAM, saving VRAM for text context.
• --reasoning off --reasoning-budget 0: Bypasses hidden thinking traces, initiating text generation within 2.08 seconds.

2. Querying Text via cURL

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-Omni-35B-A3B-SAMI-AsymQ",
    "messages": [
      {"role": "system", "content": "Răspunde scurt și precis în limba română."},
      {"role": "user", "content": "Explică pe scurt diferența dintre parametrii totali și parametrii activi într-un model MoE."}
    ],
    "temperature": 0.3,
    "stream": true
  }'

3. Querying Vision via Python

import base64
import json
import urllib.request

with open("sample.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode("utf-8")

payload = {
    "model": "Qwen3-Omni-35B-A3B-SAMI-AsymQ",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Ce conține această imagine? Răspunde în limba română."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}}
            ]
        }
    ],
    "max_tokens": 512,
    "stream": True
}

req = urllib.request.Request(
    "http://127.0.0.1:8080/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"}
)

with urllib.request.urlopen(req) as resp:
    for line in resp:
        line = line.decode("utf-8").strip()
        if line.startswith("data: ") and line != "data: [DONE]":
            chunk = json.loads(line[6:])
            content = chunk["choices"][0]["delta"].get("content", "")
            print(content, end="", flush=True)
print()

Empirical Benchmark Telemetry

Measured on an Intel Core i5-8400 (6 cores @ 2.80GHz), 32GB DDR4-2666 RAM, and NVIDIA GeForce GTX 1060 6GB (Driver 535+, Linux x86_64):

Reasoning Capability Retention (Ablation Control)

To verify that ablating 6,259 refusal columns (0.13%) and applying constrained orthogonal Procrustes stabilization preserved reasoning faculties, the base model was evaluated against SAMI-AsymQ:

The sub-0.6% delta across standard cognitive benchmarks mathematically confirms that directional feature ablation is surgical and does not induce representation collapse in the residual stream.

Disclaimer and Terms of Use

This model checkpoint is provided "as is" under the Apache-2.0 license, strictly for technical benchmarking, independent evaluation, and educational purposes.

The author (Valentin Spiridon) and TSV Consulting SRL assume no liability for direct, indirect, or consequential damages arising from the use, deployment, or outputs of this model. Users are responsible for evaluating outputs and complying with all applicable regulations.

References and Citations

If you build upon this checkpoint, methodology, or benchmarks, please cite the foundational works:

1. Directional Representation Engineering and Ablation:
   Arditi, A., et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.
2. Visual Feature Backbone (SigLIP):
   Zhai, X., et al. (2023). Sigmoid Loss for Language Image Pre-Training. arXiv:2303.15343.
3. MoE Runtime and Quantization Engine:
   Gerganov, G., et al. (2023–2026). llama.cpp: Port of Facebook's LLaMA model in C/C++ (GGUF Format & CPU-MoE Offloading). GitHub Repository.
4. Base Model Architecture:
   Qwen Team, Alibaba Cloud. (2024–2025). Qwen3-Omni: Multimodal Dense-Sparse Foundation Model. Apache-2.0.
5. Roofline Performance Model:
   Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4), 65-76.
6. SAMI Methodology, AsymQ Quantization & Technical Report:
   Spiridon, V. (2026). SAMI Omni: Asymmetric Mixture-of-Experts Topology and Memory-Decoupled Multimodal Inference on Commodity Hardware. Technical Report, TSV Consulting SRL, Timișoara, Romania. DOI: 10.5281/zenodo.23293547. ORCID: 0009-0000-2336-8527.

BibTeX Citation

@techreport{spiridon2026sami,
  title       = {SAMI Omni: Asymmetric Mixture-of-Experts Topology and Memory-Decoupled Multimodal Inference on Commodity Hardware},
  author      = {Spiridon, Valentin},
  institution = {TSV Consulting SRL},
  year        = {2026},
  month       = {October},
  doi         = {10.5281/zenodo.23293547},
  url         = {https://doi.org/10.5281/zenodo.23293547}
}