Steerling-8B:
An interpretable causal diffusion language model with concept steering.
Paper: Scaling Inherently Interpretable Language Models
Code: github.com/guidelabs/steerling
Project page: guidelabs.ai/papers/scaling-inherently-interpretable-language-models
Steerling-8B is an 8 billion parameter language model that combines masked diffusion with interpretable concept decomposition. Unlike standard autoregressive LLMs, Steerling generates text by iteratively unmasking tokens in order of confidence, and decomposes its internal representations into human-interpretable concepts that can be inspected and steered.
Quick Start
pip install steerling
from steerling import SteerlingGenerator, GenerationConfig
generator = SteerlingGenerator.from_pretrained("guidelabs/steerling-8b")
text = generator.generate(
"The key to understanding neural networks is",
GenerationConfig(max_new_tokens=100, seed=42),
)
print(text)
Model Details
| Property | Value |
|---|---|
| Parameters | 8.4B |
| Architecture | CausalDiffusionLM + iGuide |
| Context Length | 4,096 |
| Vocabulary | 100,281 (cl100k_base + specials) |
| Known Concepts | 33,732 |
| Unknown Concepts | 101,196 |
| GQA | 32 heads, 4 KV heads |
| Diff Block Size | 64 |
| Precision | bfloat16 |
| VRAM Required | ~18GB |
Architecture
Steerling uses block-causal attention, bidirectional within a block, and causal across blocks. The interpretable concept heads decompose transformer hidden states into:
hidden → known_features + unknown_features + epsilon = composed → logits
- known_features: Weighted sum of top-k learned concept embeddings (interpretable and maps to understandable features)
- unknown_features: Residual captured by a factorized unknown head (101,196 concepts, rank 256)
- epsilon: Small correction for reconstruction fidelity
Training Data
| Dataset | License | Stage |
|---|---|---|
| Nemotron-CC-HQ (real + synthetic) | NVIDIA Data Agreement | Pretraining |
| Dolmino Mix (math) | ODC-By v1.0 | Midtraining |
The Nemotron-CC dataset includes synthetic data generated by third-party models (Qwen, DeepSeek). Users should review the applicable license terms for their intended use case.
GPU Requirements
| Setup | Works? |
|---|---|
| A100 80GB | ✅ |
| A100 40GB | ✅ |
| A6000 48GB | ✅ |
| RTX 4090 24GB | ✅ |
| RTX 3090 24GB | ✅ |
| 16GB or less | ❌ |
License
The Steerling source code and model weights are released under the Apache License 2.0.
The model weights are provided for research and evaluation purposes. The weights were trained on datasets with varying license terms, including Nemotron-CC-HQ and Dolmino Mix. Some training data includes synthetic content generated by third-party models with their own license terms. We are currently reviewing the implications of these upstream licenses for downstream use of the model weights. Please check back for updates on the weight licensing terms.
For questions about commercial use of the model weights, contact us at info@guidelabs.ai