🌟 Maria-1B: Dynamic Query Adaptation (HyperGQE)

Pretrained from Scratch on 20 Billion Tokens for High-Efficiency Edge Inference

[!TIP] Key Hardware Takeaway: Under maximum sequence and batch workloads ($b=16, s=1024$), Maria-1B is 13.5% faster than Dense GQA while consuming 827 MB less peak VRAM and delivering +15.6% higher token throughput.

[!IMPORTANT] What Problem Are We Solving?
Frontier MoE models save FLOPs in datacenters, but are unusable on edge devices because all inactive experts must stay loaded in limited RAM. Standard dense models, on the other hand, burn 100% compute across every head.
Maria solves this through structured Query Sparsity: it dynamically drops 56.6M unneeded query parameters per token while using a lightweight hypernetwork to inject rich context back into active heads.

🔬 Architectural Context & Research Foundation

• 📄 Architecture Paper Preprint: https://doi.org/10.5281/zenodo.22747827
• 👨‍🔬 Author: Yadki Zahid Ahmed
• 💡 Relationship to Paper: The research preprint introduces and proves the HyperGQE architecture (originally validated on a TinyLlama-1.1B testbed). Maria-1B is the first standalone, from-scratch 20-Billion-token pretrained foundation model implementing HyperGQE with a modern 128,256 Llama-3 vocabulary, RoPE, and SwiGLU feed-forward networks.

📊 Model Specifications

⚡ Hardware Profiling & Latency Benchmarks

Measured on identical NVIDIA hardware comparing Maria (HyperGQE) against Dense GQA and Static GQE:

⏱️ Latency & Throughput Matrix

💾 VRAM & Throughput (Heavy Load: $b=16, s=1024$)

🎯 Downstream Accuracy & Representation Recovery

Static query routing (GQE) suffers from severe representational collapse because permanently pruned query heads lose attention context. Maria's dynamic hypernetwork completely eliminates this collapse:

1. Fine-Tuned Sentiment Classification (SST-2)

Maria not only recovers the 4.24% performance drop caused by static pruning—it surpasses the dense baseline while activating 102M fewer parameters.

2. Zero-Shot STEM Domain Recovery (16 Benchmarks)

🚀 Quickstart: Run Maria-1B

[!NOTE] Base Model Inference Notice: Maria-1B is a foundation pretraining base model (trained from scratch on 20B tokens of raw web text, books, and code), not an instruction-aligned chat assistant. For clean, fluent text continuation, use greedy decoding (do_sample=False) or controlled sampling (temperature=0.3, top_k=40, repetition_penalty=1.2).

[!TIP] Hardware Precision Guide:

• Google Colab Free Tier (Tesla T4) / Turing / Pascal GPUs: Use torch_dtype=torch.float16 (Tesla T4 lacks native hardware bfloat16).
• NVIDIA Ampere / Hopper / Ada (A100, H100, RTX 3090/4090): You can use either torch_dtype=torch.bfloat16 or torch.float16.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Zahid2005/Maria-1B"

# 1. Load tokenizer & model directly from Hugging Face
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,  # Recommended for Colab Tesla T4 & FP16 GPUs
    device_map="auto"
)
model.eval()

# 2. Format Prompt
prompt = "Artificial intelligence and future edge computing"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# 3. Generate with controlled base-model parameters
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=40,
        do_sample=True,
        temperature=0.3,
        top_k=40,
        repetition_penalty=1.2,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📱 Edge Computing & Mobile Deployment

Because Maria features dense FFNs and structured query-level sparsity, it avoids the massive memory penalties of MoE architectures:

• 🗜️ 4-Bit Quantization (GGUF / Q4_K_M): Compresses the model weights to ~650 MB.
• 📱 RAM Requirement: Requires only ~800 MB of total RAM, running at 45–60 tokens/sec on mobile chips (Apple A/M series, Qualcomm Snapdragon, MediaTek Dimensity, Raspberry Pi 5) with zero cloud dependencies.

📚 Research Background & Citation

The HyperGQE (Dynamic Query Adaptation) sparse attention mechanism was originally introduced and validated on a TinyLlama-1.1B backbone in:

"CAN WE IMPROVE THE PERFORMANCE OF SPARSE ATTENTION IN GROUPED QUERY EXPERTS WITHOUT INCREASING EXPERTS?"
Preprint DOI: 10.5281/zenodo.22747827

Maria-1B represents the first standalone foundation model trained from scratch for 20 Billion tokens to validate HyperGQE in an end-to-end pretraining regime.

If you use Maria-1B or the HyperGQE architecture in your work, please cite the preprint:

@article{yadki2026sparseattention,
  title={CAN WE IMPROVE THE PERFORMANCE OF SPARSE ATTENTION IN GROUPED QUERY EXPERTS WITHOUT INCREASING EXPERTS?},
  author={Yadki, Zahid Ahmed},
  journal={Zenodo},
  year={2026},
  month={September},
  doi={10.5281/zenodo.22747827},
  url={https://doi.org/10.5281/zenodo.22747827}
}