Sabi-Noema-MLA

Sabi-Noema-MLA is Sabi-Noema, our agentic model built on top of Qwen3.8-27B, converted to multi-head latent attention (MLA) for a 3.6× smaller attention cache (7.1× with an FP8 cache).

While Sabi-Noema pushes the limits of reasoning efficiency, with Sabi-Noema-MLA we see how much we can compress the memory needed per token so that long agentic sessions fit on consumer GPUs.

⚡ Performance at a Glance

• Attention Cache: 18 KiB per token (vs. 64 KiB base), 3.6× smaller | 9 KiB (7.1× smaller) with an FP8 cache
• Consumer GPUs: 7 concurrent 32K-token sessions on a 24 GB card with 4-bit weights, where the original attention fits 2 (13 with an FP8 cache)
• τ³-bench: 90.6% (vs. 79.1% base)
• Token Efficiency: 1.8× fewer output tokens per Terminal-Bench task (median 36.1K vs. 65.3K)
• Served with a vLLM plugin: sabi-mla-vllm, shipped in this repository's plugin/ directory; also loads in Transformers with trust_remote_code=True

FP8-cache results were measured with a simulated FP8 cache (cached values rounded to FP8 and read back); the released serving path stores a BF16 cache, and real FP8 cache serving is coming.

Resources

• Sabi-Noema — the original-attention model, for maximum accuracy when cache memory is not the constraint

KV Cache Compression

The MLA conversion shrinks the elements per token in the 16 attention layers, and FP8 cache quantization shrinks the bytes per element.

How the Conversion Is Done

In each of the 16 full-attention layers, the per-head key/value cache is replaced by a rank-512 compressed latent plus one 64-dimensional rotary key shared across heads, following DeepSeek-V2[1]. The rotary component is kept separate because position-dependent rotations cannot be absorbed into a low-rank projection; sharing it across heads keeps it to 64 elements per token. The new projections are fit after RL training by matching each layer's attention output to the original Sabi-Noema on long documents of up to 64K tokens. No RL is repeated, so the conversion takes about two hours on a single 8×H100 node.

Memory Requirement

Attention cache for one median-length session on each benchmark (peak prompt-plus-completion per episode, median over episodes). Reductions are against Qwen3.8-27B:

On Terminal-Bench the reduction compounds: 7.1× from the cache format, and 1.18× more because Sabi-Noema-MLA's sessions are shorter (26.0K tokens at the median with the FP8 cache against 30.6K). On the other benchmarks both models use about the same context, so the reduction stays close to the per-token figure. The median understates peak memory on Terminal-Bench: the 90th percentile is 83–95K tokens and the longest session runs to about 200K. Size deployments for the tail, not the median.

Quantizing the MLA cache to FP8 keeps scores at parity: in one paired run, every benchmark is within one task of the BF16 cache, within run-to-run noise.

Sessions per GPU. Cached tokens / concurrent 32K-token sessions:

Assumes 92% of VRAM usable, 1 GiB overhead, weights of about 16.5 / 28 / 55 GB at 4- / 8- / 16-bit, ~72 MiB of linear-attention state per session, and a single sequence capped at the model's 262K context. The 4- and 8-bit weight rows are memory arithmetic only; those weight formats are not evaluated here.

Benchmarks

Sabi-Noema-MLA (BF16 cache) against Qwen3.8-27B and comparison models, with Sabi-Noema for reference. Same harnesses for every model: Terminal-Bench 2.1 following Artificial Analysis's settings on H100 TP2 deployments with up to 8 tasks per server (Sabi-Noema-MLA, whose plugin is single-GPU, and Muse Glimmer on TP1), τ³-bench (airline, retail, telecom) with a GPT-4.1 user simulator, DeepSWE uncapped (seeds with an infrastructure failure scored as a model fail are excluded), ClawEval graded by the harness's LLM judge. Qwen reports 73.0 on Terminal-Bench 2.1 with its own Terminus setup and 42.2 on DeepSWE 1.1 with the Claude Code harness. Terminal-Bench 2.1 is averaged over 2–3 runs per open model and DeepSWE over 2–3 seeds for Sabi-Noema, Sabi-Noema-MLA and Qwen3.8-27B; GPQA-Diamond, IFBench and AA-LCR over 3 runs where we ran them. GPT-6 Luna (openai/gpt-6-luna) was called through OpenRouter at medium reasoning effort; Muse Glimmer 30B was served locally in BF16 at its highest reasoning setting with its 131K-token context, and tasks that overflowed it count as failures.

Model Overview

Quickstart

Use the same generation parameters as Qwen3.8-27B; they are the defaults in generation_config.json.

A non-thinking mode (chat_template_kwargs={"enable_thinking": false}) is supported by the chat template but was not evaluated.

vLLM (one H100 80 GB or one B300; BF16 weights and BF16 KV cache):

pip install sabi-mla-vllm      # the vLLM plugin; installs vLLM 0.31.0
vllm serve sabicap/sabi-noema-mla --max-model-len 262144 --max-num-seqs 32 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml

sabi-mla-vllm is a vLLM plugin that vLLM loads automatically; add --tensor-parallel-size 2 for two GPUs. --max-num-seqs 32: vLLM's default of 1024 exceeds the linear-attention state blocks available on one GPU. The evaluations above also used --default-chat-template-kwargs '{"preserve_thinking": false}' (earlier turns' reasoning is not replayed); examples/serve_vllm.sh runs the command with it.

Chat with any OpenAI client; tool calls use the Qwen3.8 XML format:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
tools = [{"type": "function", "function": {"name": "get_weather", "description": "Current weather for a city.",
          "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
resp = client.chat.completions.create(
    model="sabicap/sabi-noema-mla", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools,
    temperature=1.0, top_p=0.95, max_tokens=32768, reasoning_effort="xhigh", extra_body={"top_k": 20})
msg = resp.choices[0].message
print(msg.reasoning, msg.tool_calls or msg.content)

examples/ has the full serve script and a thinking/multi-turn client.

Transformers: AutoModelForCausalLM.from_pretrained("sabicap/sabi-noema-mla", dtype="bfloat16", device_map="auto", trust_remote_code=True) loads the model with the MLA layers (a reference implementation: slow, no paged cache). See examples/transformers_generate.py.

Supported: vLLM 0.31.0 on NVIDIA H100 / H200 and B300, TP 1 or TP 2. pip install sabi-mla-vllm installs it. The plugin checks the GPU and vLLM version at startup and refuses anything it hasn't been tested on (SABI_MLA_ALLOW_UNTESTED=1 overrides at your own risk). Other runtimes (llama.cpp, SGLang, Ollama) can't run the MLA layers yet: use Sabi-Noema there.

Checking that the plugin is active. At startup vLLM prints, once per GPU:

[sabi-mla] H100/H200 (sm_90): prefill kernel fa3, decode kernel fa3 (vLLM 0.31.0, TP 1)
[sabi-mla] shared-rotary MLA active on 16 layers [3, 7, ..., 63] | cache width 576 (latent 512 + shared rotary 64) vs BF16 2048 | artifact 3b4512c1...
[sabi-mla] self-check PASS on H100/H200 (sm_90): decode (fa3) rel err ... prefill (fa3) rel err ...

Without the plugin, vLLM refuses the model (this repository declares its own model type, sabi_mla) instead of serving it with the original attention. Transformers without trust_remote_code=True refuses it for the same reason.

Troubleshooting

• sabi-mla-vllm: vLLM X is untested / GPU ... is untested: install a supported vLLM (pip install sabi-mla-vllm picks one) or use a listed GPU; SABI_MLA_ALLOW_UNTESTED=1 overrides.
• Tool calls come back as plain text: serve with --enable-auto-tool-choice --tool-call-parser qwen3_xml (not hermes).
• --kv-cache-dtype fp8 is not supported: FP8 cache serving is not implemented yet; the FP8 results above use a simulated cache. Serve with the default BF16 cache.
• llama.cpp, SGLang, Ollama: not supported for this model yet; use Sabi-Noema.

Use Cases

• Many concurrent agent sessions per GPU: 13 32K-token sessions on a 24 GB card, 50 on an H100, against 2 and 8 with the original attention (FP8 cache; 7 and 27 with today's BF16 cache)
• Long-context agents on consumer hardware: half a million tokens of cache on an RTX 4090 with 4-bit weights and an FP8 cache
• Policy-bound customer agents at scale: τ³-bench at 90.6%, 11.5 points ahead of Qwen3.8-27B, where throughput per GPU sets the serving cost
• Cost-sensitive agentic serving: ahead of Qwen3.8-27B on τ³-bench, ClawEval and DeepSWE, at a seventh of the cache and about half the output tokens on terminal work

Limitations

• The conversion costs accuracy on the longest tasks: against Sabi-Noema, Terminal-Bench 2.1 falls by about 10 points, DeepSWE by about 8 points and MCPMark by 14 tasks; when cache memory is not the constraint, use Sabi-Noema
• Trails Qwen3.8-27B on terminal and MCP tool work: Terminal-Bench 2.1 65.9% against 73.0%, MCPMark 34 against 36 of 55, τ³-bench banking 36.5% against 39.8%
• Knowledge and long context trail Qwen3.8-27B: GPQA-Diamond −2.2, AA-LCR −2.7
• τ³-bench conversations run longer: 5.0K tokens against 4.2K for Qwen3.8-27B, the one benchmark where the MLA model writes more
• Serving needs the plugin: vLLM only, on the versions and GPUs listed under Quickstart; real FP8 cache serving is not available yet
• Run-to-run noise: differences of a task or two are within noise

Contact

contact@sabi.com

References

[1] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024.