Susono-10B-A1B-Base

English | 日本語

Susono-10B-A1B-Base is an original-architecture LLM with 10B total parameters and about 1B active parameters per token (A1B). It integrates Engram (a conditional memory module) and mHC-lite (Manifold-Constrained Hyper-Connections Lite) into a hybrid backbone of Full Attention + GatedDeltaNet + MoE.

Training was performed on the NVIDIA GH200 Grace Hopper Superchip. Dedicated fused kernels were implemented for Engram and mHC-lite, and training was optimized with FP8 training + CPU offload, taking advantage of the GH200 GPU architecture.

Note that this model was developed purely as a personal hobby project and funded privately. The development cost was only about USD 1,875 (roughly JPY 300,000), so please be aware that pre-training and post-training have not been carried out to a sufficient extent.

⚠️ This is a pre-trained base model. It has not been fine-tuned for instruction following or chat use, and is not intended to be used directly as a conversational assistant.

We assume no responsibility for the model's outputs. Use it at your own risk.

Model Overview

Reference papers:

• Engram: arXiv:2601.07372v1 "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models"
• mHC-lite: arXiv:2601.05732v1 "mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations"

Architecture

• Full Attention + GatedDeltaNet: A hybrid configuration that uses full softmax attention every 4 layers (full_attention_interval=4) and GatedDeltaNet (linear attention) in the remaining layers.
• Sparse MoE: All FFN layers are MoE (96 experts, 4 active per token).
• Engram (conditional memory): O(1) lookup into static embeddings via N-gram hashing. It directly retrieves local, repetitive patterns and frees up attention for global-context processing. Inserted at layers 3 and 7, it serves as the primary store of factual knowledge.
• mHC-lite (multi-stream residual connections): Dynamic residual connections across multiple streams. Leveraging the Birkhoff–von Neumann theorem, it strictly guarantees a doubly stochastic matrix without any Sinkhorn-Knopp iterations.

Training Environment

NVIDIA GH200 Grace Hopper Superchip

The GH200 is a heterogeneous superchip that directly connects a Grace CPU (Arm Neoverse V2 / 72 cores) and a Hopper GPU (H100-class / 96GB HBM3) via NVLink-C2C (900GB/s bidirectional, 7× the bandwidth of PCIe Gen5). Hardware-level memory coherency lets the CPU and GPU access each other's memory without page migration, making full-scale CPU offload practical.

Training Framework

Based on Megatron-LM, extended for Susono as follows:

• Triton Fused Kernels: Fuse operations such as Engram lookup, mHC width connection, GatedDeltaNet decay, MoE router, RMSNorm variants, aux loss, and cross entropy. Every kernel includes a PyTorch fallback.
• FP8 training + CPU offload: Parameters are kept in FP8 (e4m3), while the Adam optimizer state and master weights (BF16) are offloaded to CPU memory over NVLink-C2C.

Training Schedule

Usage

As this is a base model, use it for text completion (continuation). Do not apply a chat template.

This model relies on a custom architecture, so you need to install the dedicated transformers fork (puwaer/transformers) before running inference.

git clone https://github.com/puwaer/transformers.git
cd transformers
pip install -e .

After installing, run inference as follows:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "puwaer/Susono-10B-A1B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "A large language model is"
model_inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=16384,
    do_sample=True,
    temperature=0.2,
    top_p=0.9,
    repetition_penalty=1.05,
)
output = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(output)

If you want to serve this model with vLLM or SGLang, use the dedicated forks puwaer/sglang and puwaer/vllm instead (see the branches listed in Reference Repositories below).

Reference Repositories

• HuggingFace transformers implementation: https://github.com/puwaer/transformers.git (main branch)
• Megatron-LM implementation: https://github.com/puwaer/Megatron-LM.git (main branch)
• SGLang implementation: https://github.com/puwaer/sglang.git (sglang-v0.5.10-add-suson-model branch)
• vLLM implementation: https://github.com/puwaer/vllm.git (vllm-v0.19.1-add-suson-model branch)