Maba v1.5 (103.5M)
Reference research proof-of-concept for the Maba v1.5 experimental architecture.
- Base Architecture: AndrewThompson1233/maba-v1.5-exp-architecture
- Parameters: 103,520,911 (103.5M)
- Core Computation Ratio: 95.21% (4.30% Vocab Tax)
- Macro-Stack: 3:1 (15 DGDA Recurrence : 5 MABA-SA Dynamic Sparse Attention)
- Positional Encoding: Strict NoPE (0 parameters)
- Training Corpus: 3,044 dialogue pairs on NVIDIA L4 (bfloat16)
Empirical Benchmark vs Qwen3.8-Flash-Next (101.7M)
Evaluated under identical training budgets (3,044 dialogues, 15 epochs, bfloat16, NVIDIA L4):
| Metric | Maba v1.5-exp | Qwen3.8-Flash-Next | Delta / Advantage |
|---|---|---|---|
| Parameters | 103,520,911 (103.5M) | 101,701,120 (101.7M) | 0.2% parity |
| Architecture | 75% DGDA + 25% MABA-SA | 75% GDN + 25% QSA + MoE | Cyclic 3:1 |
| Positional Encoding | Strict NoPE (0 params) | 25% Partial RoPE | Zero positional overhead |
| Contrastive Retrieval (MCQ) | 87.5% (7/8) | 75.0% (6/8) | +12.5% accuracy |
| Validation Loss | 0.0697 | 0.0778 | -10.4% entropy |
| Validation Perplexity (PPL) | 1.07 | 1.08 | Maba wins |
| Decode Throughput (L4) | 7.0 tok/s | 5.5 tok/s | +27.3% faster generation |
Attention Ablation Proof
Empirical demonstration of the contribution of the 25% MABA-SA dynamic sparse attention layers against a pure linear recurrent baseline on the exact same checkpoint weights:
| Model Variant | Attention Mechanism | Validation Loss | Perplexity (PPL) | Error Reduction |
|---|---|---|---|---|
| Pure DGDA (Ablation) | None (100% Linear Recurrence) | 3.9360 | 51.21 | Baseline |
| Qwen3.8-Flash-Next | QSA (GQA + Micro-block Indexer) | 3.8772 | 48.29 | -5.7% vs Recurrence |
| Maba v1.5 Full | MABA-SA (MLA + Top-32 + HCA) | 3.5903 | 36.24 | -29.2% error drop |
Needle-In-A-Haystack & Centroid Retrieval (512 to 4096 Tokens)
| Context Length | Needle Position | Needle Block | DG-Indexer (Hybrid Mean+Max) | Standard Pure Mean Pooling |
|---|---|---|---|---|
| 512 tokens | 51 (10%) | Block #0 | Retrieved (Top-32) | Retrieved |
| 512 tokens | 256 (50%) | Block #4 | Retrieved (Top-32) | Retrieved |
| 512 tokens | 460 (90%) | Block #7 | Retrieved (Top-32) | Retrieved |
| 1024 tokens | 102 (10%) | Block #1 | Retrieved (Top-32) | Retrieved |
| 1024 tokens | 512 (50%) | Block #8 | Retrieved (Top-32) | Retrieved |
| 1024 tokens | 921 (90%) | Block #14 | Retrieved (Top-32) | Retrieved |
| 2048 tokens | 204 (10%) | Block #3 | Retrieved (Top-32) | Retrieved |
| 2048 tokens | 1024 (50%) | Block #16 | Retrieved (Top-32) | Retrieved |
| 2048 tokens | 1843 (90%) | Block #28 | Retrieved (Top-32) | Retrieved |
| 4096 tokens | 2048 (50% Lost-in-Middle) | Block #32 | Retrieved (Top-32) | Diluted to 0.0 (Failed) |
| 4096 tokens | 3686 (90%) | Block #57 | Retrieved (Top-32) | Retrieved |
KV-Cache Footprint at 4k Context
| Context Length | Dense Attention (Baseline) | Qwen3.8-Flash-Next | Maba v1.5 (MLA + Top-32) | Memory Reduction vs Dense |
|---|---|---|---|---|
| 512 tokens | 25.00 MB | 1.00 MB | 0.62 MB | -97.5% |
| 1,024 tokens | 50.00 MB | 2.00 MB | 1.25 MB | -97.5% |
| 2,048 tokens | 100.00 MB | 4.00 MB | 2.50 MB | -97.5% |
| 4,096 tokens | 200.00 MB | 8.00 MB | 2.50 MB | -98.8% |
How to Use
Load the base architecture from AndrewThompson1233/maba-v1.5-exp-architecture and load the weights:
import sys
import subprocess
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
if "maba-v1.5-exp-architecture" not in sys.path:
subprocess.run(["git", "clone", "https://huggingface.co/AndrewThompson1233/maba-v1.5-exp-architecture"], check=False)
sys.path.insert(0, "maba-v1.5-exp-architecture")
from maba_sparse.config import MabaSparseConfig
from maba_sparse.model import MabaSparseForCausalLM
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
cfg = MabaSparseConfig(vocab_size=len(tokenizer), dim=640, d_emb=128, intermediate_size=1248, n_layers=20)
model = MabaSparseForCausalLM(cfg).to(device)
weights_path = hf_hub_download(repo_id="AndrewThompson1233/maba-1.5-103m", filename="model.safetensors")
weights = load_file(weights_path)
model.load_state_dict(weights)
model.eval()
prompt = "User: tell me a joke\nAssistant: "
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)
with torch.no_grad():
with torch.amp.autocast(device_type="cuda", dtype=torch.bfloat16):
output_ids = model.generate(input_ids, max_new_tokens=35, temperature=0.0)
print(tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True).strip())