Nool-Alpha-100M-Chat
Nool-Alpha-100M-Chat is an efficient bilingual (Indonesian & English) and Python code causal language model based on an innovative hybrid architecture:
- GSLA (Grouped-Subspace Latent Attention): Compresses KV-cache by 87.8% compared to standard dense MHA.
- HFK-MoE (Heterogeneous Factorized MoE): Dense SwiGLU shared anchor + 8 low-rank factorized experts (r=96, Top-2 routing) saving 21.3% FLOPs.
- Global Residual Highway: Stabilizes deep residual pathways with $\tanh(\alpha) \cdot \text{RMSNorm}(x_0)$.
- Logit Soft-Capping: 30.0 tanh threshold preventing logit divergence.
Official Code Repository: GitHub - Ch3nOff/Nool-Alpha
Model Details
- Architecture:
nool_alpha - Total Parameters: ~149.8M (~111M)
- Active Parameters / Token: ~97.9M
- Vocabulary Size: 50257 (GPT-2 BPE)
- Context Length: 2048 tokens
- Training Step: 900
- Recorded Loss: 2.633549153804779
Files Included
model.safetensors: Model weights in safe, zero-copy format.config.json: Complete architectural hyper-parameters.- Tokenizer files:
vocab.json,merges.txt,tokenizer.json, etc.
Quick Start (PyTorch)
import torch
from safetensors.torch import load_file
from nool_alpha.config import NoolAlphaConfig
from nool_alpha.model import NoolAlphaForCausalLM
# Load configuration and weights
config = NoolAlphaConfig.from_dict({
"architectures": [
"NoolAlphaForCausalLM"
],
"model_type": "nool_alpha",
"vocab_size": 50257,
"d_model": 768,
"n_layers": 10,
"num_heads": 12,
"head_dim": 64,
"d_c": 192,
"d_pe": 32,
"rope_theta": 500000.0,
"max_position_embeddings": 2048,
"sliding_window": 512,
"swa_interval": 4,
"shared_ffn_dim": 1536,
"num_experts": 8,
"top_k_experts": 2,
"expert_rank": 96,
"moe_aux_loss_coeff": 0.01,
"highway_alpha_init": 0.05,
"logit_soft_cap": 30.0,
"rms_norm_eps": 1e-06,
"tie_word_embeddings": true,
"checkpoint_step": 900,
"checkpoint_loss": 2.633549153804779
})
model = NoolAlphaForCausalLM(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()