gdn-340m-isp-hybrid-3to1-10b
This is a 340M controlled-pretraining checkpoint released for the paper Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus.
It is one member of the Massive Activations HLA model release and the corresponding Hugging Face Collection. The official analysis and reproducibility code is available at StartluxLabs/Massive-Activations-HLA.
Checkpoint details
| Field | Value |
|---|---|
| Model | gdn-340m-isp-hybrid-3to1-10b |
| Scale | 340M |
| Training tokens | 10B |
| Experiment | ISP |
| Full-attention layers | 3, 6, 9, 12, 15, 18, 21, 24 (one-based) |
| Output-gating variant | GDN output gate enabled; full-attention output gate disabled |
| Final training step | 19073 |
| Weight format | Safetensors |
Compatibility and reproducibility scope
This checkpoint loads with the public, pinned environment documented in the GitHub repository:
conda create -n ma-hla python=3.12 -y
conda activate ma-hla
bash scripts/install_released_gdn_cu126.sh
Public FLA is pinned to v0.5.2, commit
9c8e42e762fce087c27b673af4922795d9edb85e. Exact A800/CUDA 12.6 package
versions are recorded in requirements/released-gdn-cu126.txt.
Loading
REPO_ID = "startlux-models/gdn-340m-isp-hybrid-3to1-10b"
Register the public FLA architecture before using Transformers directly:
import fla.models.gated_deltanet # registers the custom config/model
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForCausalLM.from_pretrained(
REPO_ID,
torch_dtype="auto",
)
For analysis, prefer the GitHub registry and scripts because they validate the FLA version and recover full-attention layer metadata consistently.
Reproduction
Use the official GitHub repository for the tested analysis workflow, PAS/ISP metrics, lifecycle atlases, installation instructions, and model registry:
Intended use and limitations
This checkpoint is a research artifact for studying massive activations, hybrid linear-attention architectures, PAS/ISP morphology, attention placement, output gating, and scale. It is not instruction-tuned, safety-tuned, or validated for production deployment. It has not been comprehensively evaluated for downstream accuracy, factuality, bias, robustness, privacy, or safety.
The checkpoint uses the custom Transformers architecture
GatedDeltaNetForCausalLM (model_type="gated_deltanet").
Training data
The model was trained from scratch on open data. A dataset-level composition and sampling breakdown is not included in this release; users should not infer language or domain coverage beyond the published project materials.
Citation
Please cite the accompanying paper. The arXiv link and final BibTeX entry will be added after the preprint metadata is public.
License
The released model artifacts are available under the Apache License 2.0. See
LICENSE. Third-party software and datasets retain their own
licenses and terms.