Sonexa-Beta
Sonexa-Beta is a lightweight, bilingual (English / Russian) language model powered by a custom Mixture of Experts (MoE) architecture. Built from scratch for ultra-low latency inference, edge devices, and consumer hardware.
- Developer / Organization: Sonexa Artificial Intelligence
- Model Architecture: Causal Language Model with Mixture of Experts (MoE)
- License: Apache-2.0
Architectural Specifications
The following metrics are dynamically extracted directly from the model weights:
| Metric | Measured Value |
|---|---|
| Total Parameters | 140,933,632 (140.93M) |
| Active Parameters (All-Experts / Soft MoE) | 140,933,632 (140.93M) |
| Active Parameters (Top-1 MoE Mode) | 70,154,752 (70.15M) |
| Tied Embedding Weights | 8,192,000 (8.19M) |
| Attention Layers (GQA) | 6,558,720 (6.56M) |
| Shared Expert Capacity | 23,592,960 (23.59M) |
| Routed Experts Capacity (4x) | 94,371,840 (94.37M) |
| Hidden Size | 512 |
| Intermediate Size (SwiGLU) | 1536 |
| Attention Mechanism | Grouped Query Attention (8 Query Heads / 2 KV Heads) |
| Positional Encoding | Rotary Position Embeddings (Real Cos/Sin RoPE) |
| Vocabulary Size | 16,000 (Byte-Level BPE) |
| Context Window | 256 tokens |
| Memory Footprint (FP16) | ~268.81 MB VRAM |
| Release Artifact Size | 537.64 MB |
Key Characteristics
- Bilingual Dialogue: Trained on a balanced bilingual corpus combining human-curated English dialogues (
HuggingFaceH4/no_robots) and filtered Russian conversational datasets. - Anti-Refusal Calibration: Calibrated via output projection damping to eliminate generic AI assistant disclaimers ('As an AI language model...', 'I do not have personal opinions...').
- Soft MoE Routing: Concurrent all-expert evaluation dynamic weighting for consistent knowledge routing and response quality.
- Extreme Portability: Fits comfortably in under 300 MB of memory in FP16, allowing fast CPU and consumer GPU execution.
Prompt Template
Sonexa-Beta expects inputs formatted as follows:
<|user|>
Your message here
<|bot|>
Model response here<|endoftext|>
Citation
@misc{sonexa2026,
author = {Sonexa Artificial Intelligence},
title = {Sonexa-Beta: Lightweight Bilingual Mixture-of-Experts Language Model},
year = {2026},
publisher = {Hugging Face},
journal = {Hugging Face Model Hub},
howpublished = {\url{[https://huggingface.co/](https://huggingface.co/)Cartik/Sonexa-Beta}}
}