Aether-6B-11Attn
⚠️ Pretraining is still in progress. This repository currently holds a mid-training checkpoint (step 174,000); it will be updated with the final weights as soon as training completes. The
-basesuffix denotes "no instruction tuning" — an-itvariant will follow once pretraining is finished.
Aether family — 
Which Aether model should I use?
| Model | What it is | Pick it if |
|---|---|---|
| Aether-7B-5Attn | 6.59B MoE base. Fully open - weights + data recipe + training code + all 162k-step logs + checkpoints | You want to audit, verify or rebuild a foundation model end to end |
| Aether-7B-5Attn-it | The same model, instruction-tuned | You want it to answer rather than continue text |
| AETHER-7B-7Attn-base | Same 49-layer architecture, a different checkpoint. Open weights | You want a second run of this architecture to compare against |
| Aether-6B-11Attn-base | 121 layers, 11 sequence-mixing mechanisms in one network - attention, Mamba-2, Hyena, GDN, MLA - on an 11x11 Latin square | You research heterogeneous sequence mixing. It is a mid-training research artifact |
All four load the same way:
AutoModelForCausalLM.from_pretrained(MODEL, trust_remote_code=True, dtype=torch.bfloat16)
Layer map — 11x11 Latin square
Eleven sequence-mixing mechanisms across 121 layers, each appearing exactly once per row and per column. Colour marks the family: attention (blue), linear (teal), state-space/Mamba2 (orange), convolutional/Hyena (purple) — the non-attention families are what make this heterogeneous rather than a variant sweep.
A 121-layer network that places 11 different sequence-mixing mechanisms on an 11x11 Latin square.
To our knowledge, this is the first model to integrate 11 heterogeneous sequence-mixing mechanisms — attention, state-space, convolutional and linear families — inside a single network. Existing hybrids combine two or three (e.g. Mamba + Transformer).
Release scope: open weights. This repository ships the weights plus the minimum code needed to load them. Unlike Aether-7B-5Attn — which is fully open (data recipe, training code, logs, checkpoints) — the training data recipe, training code, logs and intermediate checkpoints for this model are not released.
The 11 mechanisms
121 layers = 11 x 11. Each mechanism occupies exactly 11 layers, one per row and column of the Latin square, so no mechanism concentrates at any depth.
| # | Mechanism | Family | Distinctive parameters (verifiable in the weights) |
|---|---|---|---|
| 0 | nsa | sparse attention | compress_mlp, gate_logit, select_score |
| 1 | differential | attention | lambda_q1/k1/q2/k2, subln |
| 2 | full | attention | q/k/v/o_proj |
| 3 | mla | latent attention | kv_a_proj, kv_b_proj, q_a_proj, q_b_proj |
| 4 | sliding | attention | windowed masking over the full path |
| 5 | compress | attention | full_attn.* submodule |
| 6 | hybrid | attention | nsa.* + diff.* + gate + merge_norm |
| 7 | linear | linear attention | gate, norm |
| 8 | mamba2 | state-space | A_log, D, conv1d, dt_proj, B_proj, C_proj |
| 9 | gdn | gated delta net | conv1d_q, conv1d_k, gate_proj |
| 10 | hyena | convolutional | filter_dense (implicit filter) |
Precision matters here: full and sliding share the same parameters (sliding is masking over the full path), so the weights show 10 distinct parameter signatures for 11 labels. And several of these are not "attention" in the strict sense — mamba2, hyena, gdn and linear are other sequence-mixing families, which is exactly the point of the experiment.
Specifications
| Parameters | 5.79B total (MoE) |
| Layers | 121 (11 x 11 Latin square) |
| Hidden size | 1024 |
| Attention heads | 8 query / 2 key-value (GQA), head dim 128 |
| Experts | 25 routed (top-7) + 1 shared |
| Vocabulary | 151,936 |
| Precision | bfloat16 |
| Checkpoint | training step 174,000 |
| License | Apache-2.0 |
Provenance and honest caveats
This model was reconstructed from a sharded FSDP training checkpoint. The exact training code was not archived, so the architecture code shipped here was rebuilt and then validated two ways:
- Exact parameter match — all 11,047 checkpoint tensors load with 0 missing / 0 unexpected keys, and the parameter count matches the checkpoint exactly (5.7885B).
- Perplexity — the loaded model predicts real text sensibly (PPL 11.8 English / 6.5 Korean on sample sentences). A mis-reconstructed architecture would produce perplexity orders of magnitude worse.
Treat this as a research artifact, not a polished product: it is a mid-training checkpoint of an architecture experiment, with no instruction tuning.
Usage
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL = "FINAL-Bench/Aether-6B-11Attn-base"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
MODEL, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
).eval()
trust_remote_code=True is required — this is a custom architecture (aether_v2_11attn)
and the modeling code ships in this repository.
Run at batch_size = 1 (the NSA branches do not consume a padding mask), and run the forward under
torch.autocast(dtype=torch.bfloat16)— several branches create fp32 intermediates.
Greedy decoding degenerates into repetition (expected for an un-tuned base checkpoint) — use sampling with a repetition penalty. Generation is ~10 s/token: 121 layers with no KV cache. This is a research artifact, not a serving build.
Related
- Fully-open sibling: Aether-7B-5Attn (weights + data recipe + training code + logs + checkpoints)
- Collection: Aether Foundation Model
Built by VIDRAFT. License: Apache-2.0