gdn-gatedfa-340m-isp-hybrid-3to1-10b
This is a 340M controlled-pretraining checkpoint released for the paper Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus.
It is one member of the Massive Activations HLA model release and the corresponding Hugging Face Collection. The official analysis and reproducibility code is available at StartluxLabs/Massive-Activations-HLA.
Checkpoint details
| Field | Value |
|---|---|
| Model | gdn-gatedfa-340m-isp-hybrid-3to1-10b |
| Scale | 340M |
| Training tokens | 10B |
| Experiment | ISP |
| Full-attention layers | 3, 6, 9, 12, 15, 18, 21, 24 (one-based) |
| Output-gating variant | GDN and full-attention output gates enabled |
| Final training step | 19073 |
| Weight format | Safetensors |
Compatibility and reproducibility scope
The checkpoint is a weights-only research artifact. Its full-attention layers use a post-SDPA, head-specific sigmoid output gate inspired by the G1 design in Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (official code). The exact GatedDeltaNet integration is not distributed, so this checkpoint is not part of the public from-scratch quickstart. The repository loader raises an explicit compatibility message unless a user deliberately configures a compatible local implementation.
Loading
REPO_ID = "startlux-models/gdn-gatedfa-340m-isp-hybrid-3to1-10b"
The files can be downloaded without a Hugging Face subfolder argument:
from huggingface_hub import snapshot_download
local_path = snapshot_download(REPO_ID)
Downloading the weights does not make this variant directly executable; see the compatibility scope above.
Reproduction
Use the official GitHub repository for the tested analysis workflow, PAS/ISP metrics, lifecycle atlases, installation instructions, and model registry:
Intended use and limitations
This checkpoint is a research artifact for studying massive activations, hybrid linear-attention architectures, PAS/ISP morphology, attention placement, output gating, and scale. It is not instruction-tuned, safety-tuned, or validated for production deployment. It has not been comprehensively evaluated for downstream accuracy, factuality, bias, robustness, privacy, or safety.
The checkpoint uses the custom Transformers architecture
GatedDeltaNetForCausalLM (model_type="gated_deltanet").
Training data
The model was trained from scratch on open data. A dataset-level composition and sampling breakdown is not included in this release; users should not infer language or domain coverage beyond the published project materials.
Citation
Please cite the accompanying paper. The arXiv link and final BibTeX entry will be added after the preprint metadata is public.
License
The released model artifacts are available under the Apache License 2.0. See
LICENSE. Third-party software and datasets retain their own
licenses and terms.