Ael-Coder-16B-MoE

15.71B Total / 2.36B Active Mixture-of-Experts Code Intelligence • ISOM-R2 Bounded-Memory Architecture

Overview

Ael-Coder-16B-MoE is a 15.71-billion-parameter Mixture-of-Experts code intelligence model (64 routed experts, 2 shared experts, 2.36B active parameters per token) equipped with Multi-Head Latent Attention (MLA, $d_k = 192$) and powered by the ISOM-R2 (Isometric State Operator Manifold) multi-million-token architecture.

Designed for repository-scale code synthesis and cross-module dependency resolution, Ael-Coder-16B-MoE streams 2,774,027 real codebase tokens (1,355 chunks across 82 source files from huggingface/transformers) in 12.18 seconds (227,686 tokens/sec) on a single NVIDIA A100-40GB GPU. Throughout the entire 2.77M-token prefill, active KV cache length remains locked at 2,112 tokens with only +0.06 GB streaming VRAM overhead (29.28 GB model weights $\rightarrow$ 29.34 GB streaming VRAM, 30.51 GB peak VRAM during synthesis).

Core Architectural Capabilities

1. Hierarchical Paged Virtual SVD Cache (ISOMR2VirtualSVDCache)
   Streams multi-million-token repositories in 2,048-token chunks across MLA decoder layers while strictly bounding active GPU KV memory to 2,112 tokens (64 attention sinks + 2,048 rolling window).
2. Entity-Balanced Multi-Hop Retrieval (ISOMR2Engine)
   Automatically intercepts long-context inputs inside model.generate(), balancing per-entity lexical and spectral recall to isolate exact target definitions separated by over 1.39 million tokens ([339, 1020], zero distractors).
3. $SO(192)$ Multi-Head Latent Attention Lie-Manifold Transport
   Enforces skew-symmetric Cayley orthogonal transport ($R = (I - \frac{1}{2}A)^{-1}(I + \frac{1}{2}A) \in SO(192)$) across the 192-dimensional MLA head space, achieving an audited orthogonality error of $4.47 \times 10^{-5}$ and an $O(1)$ multi-hop state rollback error of $3.06 \times 10^{-6}$.

Ael Model Family — Audited Multi-Million-Token Benchmarks

All four models in the Ael benchmark suite are evaluated on unpadded, real-world open-source repositories (huggingface/transformers, sympy/sympy, and django/django) on NVIDIA A100-SXM4-40GB hardware. Interactive comparisons and telemetry logs are hosted on the Official Ael Benchmark Space.

Audited Benchmark 1: 2,774,027-Token Multi-File MoE Code Synthesis

Executed Notebook: Ael_Coder_16B.ipynb

Task Methodology

Evaluated on 82 real Python source files (9,779,186 characters, 2,774,027 real tokens across 1,355 chunks) from huggingface/transformers on an NVIDIA A100-SXM4-40GB GPU. The model must retrieve two target classes separated by 1,394,629 real tokens (681 chunks apart):

• Hop 1 (Chunk 339, Token #695,058): class CaptureStdout(CaptureStd) in testing_utils.py
• Hop 2 (Chunk 1020, Token #2,089,687): class GELUActivation(nn.Module) in activations.py

Using both retrieved definitions, the model synthesizes a LoggedGELU(GELUActivation) class that captures standard output during the forward pass and is verified live on GPU against PyTorch's native GELUActivation.

Hardware Execution Log (NVIDIA A100-SXM4-40GB)

Loaded AelCoder16BMoEForCausalLM (15.71B total / 2.36B active MoE params) | Weights VRAM: 29.28 GB
Tokenizing 2.25M+ real-world multi-file corpus (82 files | 9,779,186 chars)...
Hop 1 Target   : class CaptureStdout(CaptureStd) @ Token #695,058 -> Chunk 339
Hop 2 Target   : class GELUActivation(nn.Module) @ Token #2,089,687 -> Chunk 1020
Inter-File Gap : 1,394,629 real tokens (681 chunks apart)

  [ISOM-R2 Engine] Streaming 2,773,953 codebase context tokens across 1355 chunks (active GPU buffer < 400 MB)...
  [ISOM-R2] Prefill 552,960 / 2,773,953 tokens (19.9%) | Active buffer: 2112 tokens | VRAM: 29.34 GB
  [ISOM-R2] Prefill 1,105,920 / 2,773,953 tokens (39.9%) | Active buffer: 2112 tokens | VRAM: 29.34 GB
  [ISOM-R2] Prefill 1,658,880 / 2,773,953 tokens (59.8%) | Active buffer: 2112 tokens | VRAM: 29.34 GB
  [ISOM-R2] Prefill 2,211,840 / 2,773,953 tokens (79.7%) | Active buffer: 2112 tokens | VRAM: 29.34 GB
  [ISOM-R2] Prefill 2,764,800 / 2,773,953 tokens (99.7%) | Active buffer: 2112 tokens | VRAM: 29.34 GB
  [ISOM-R2] Prefill 2,773,953 / 2,773,953 tokens (100.0%) | Active buffer: 2112 tokens | VRAM: 29.34 GB
  [ISOM-R2] Retrieved salient context pages: [339, 1020] | Active KV: 2112 tokens

============================================================================================
2.77M-TOKEN MULTI-FILE MOE SYNTHESIS BENCHMARK: Prannesshkva/Ael-Coder-16B-MoE
============================================================================================
Total Real Context Tokens : 2,774,027 (82 files | 1355 chunks)
Inter-File Hop Distance   : 1,394,629 tokens (Chunk 339 <-> Chunk 1020)
Total Time (Prefill+Gen)  : 12.18 s (227,686 tok/s)
Model Weights VRAM        : 29.28 GB
Peak Total GPU VRAM       : 30.51 GB (KV + Activation Overhead: +1.23 GB)
Active KV Cache Length    : 2112 tokens
Retrieved Chunk Indices   : [339, 1020]
Multi-Hop Recall          : Hop 1 Hit: True | Hop 2 Hit: True
SO(192) MLA Orthogonality : 4.47e-05 ||R^T R - I||_F  |  O(1) Rollback Error: 3.06e-06
Live Execution Verification: PASSED (Output shape: [1, 4], Captured Stdout: 'torch.Size([1, 4])', max_err = 0.00e+00)
============================================================================================

Audited Benchmark 2: 1,089,849-Token Single-Hop Repository Retrieval

Evaluated on 93 real Python source files (3,867,627 characters, 1,089,849 tokens across 533 chunks) from huggingface/transformers and pytorch/pytorch:

Loaded AelCoder16BMoEForCausalLM | Model VRAM: 31.49 GB
Corpus: 93 real files | 3,867,627 chars | 1,089,849 tokens | Ground-Truth: Chunk 257
  [ISOM-R2 Engine] Streaming 1,089,795 codebase context tokens across 533 chunks (active GPU buffer < 400 MB)...
  [ISOM-R2] Prefill 1,089,795 / 1,089,795 tokens (100.0%) | Active buffer: 2112 tokens | VRAM: 31.49 GB
  [ISOM-R2] Retrieved salient context pages: [258, 257] | Active KV: 2112 tokens

================================================================================
BENCHMARK RESULTS: Prannesshkva/Ael-Coder-16B-MoE
================================================================================
Total Real Context Tokens : 1,089,849
Total Time (Prefill+Gen)  : 15.86 s (68,717 tok/s)
Model Weights VRAM        : 31.49 GB
Peak Total GPU VRAM       : 32.06 GB ( Overhead: +0.57 GB )
Active KV Cache Length    : 2112 tokens
Retrieved Chunk Indices   : [258, 257] (Exact Token Ground-Truth: Chunk 257)
Ground-Truth Base Class   : CaptureStd
Model Generated Output    : CaptureStd
================================================================================

Architectural Specifications

Quickstart Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Prannesshkva/Ael-Coder-16B-MoE"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

prompt = """<|im_start|>user
Write a high-performance concurrent queue in Python using lock-free atomic operations.<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
        tokenizer=tokenizer,
    )

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Citation, Dual-Layer Licensing & Upstream Attribution

@software{ael_coder_16b_moe_2026,
  author    = {Prannessh K. V. A.},
  title     = {Ael-Coder-16B-MoE: Multi-Million-Token Repository-Scale Bounded-Memory MoE Powered by ISOM-R2},
  year      = {2026},
  publisher = {CERN Zenodo},
  doi       = {10.5281/zenodo.22649142},
  url       = {https://huggingface.co/Prannesshkva/Ael-Coder-16B-MoE}
}

Dual-Layer License Structure (DeepSeek License Section 3 & MIT Compliance)

Pursuant to Section 3 of the DeepSeek License Agreement and the MIT License, this repository separates licensing between the unmodified upstream pretrained foundation weights and the author's original architectural modifications:

Mandatory Upstream Notice & Trademark Disclaimer (DeepSeek License Sections 3 & 4)

"DeepSeek-Coder-V2 is licensed under the MODEL LICENSE. Copyright (c) 2024 DeepSeek. All Rights Reserved."

1. Prominent Notice of Modification (Section 3(b)): Modified by Prannessh K. V. A. to integrate the ISOM-R2 (Isometric State Operator Manifold) Paged Virtual SVD Cache (ISOMR2VirtualSVDCache), entity-balanced multi-hop codebase retrieval (ISOMR2Engine), $SO(192)$ Multi-Head Latent Attention Lie-group transport, and AelCoder16BMoEForCausalLM execution bindings. Full modification logs and Attachment A Use-Based Restrictions are included in LICENSE and NOTICE.
2. Distinct Naming & Non-Endorsement (Section 4 — Trademarks): In compliance with Section 4 of the DeepSeek License Agreement, this derivative architecture is published under the distinct Ael name (Ael-Coder-16B-MoE) so as not to imply endorsement by or affiliation with the original licensor. DeepSeek is a trademark of DeepSeek AI. This independent research work is not affiliated with, sponsored by, or endorsed by DeepSeek AI.