Graphite Alpha 26B-A4B
Graphite Alpha is an experimental 26B-class sparse Mixture-of-Experts language model with approximately 4B active parameters per token, released as a W4A16 QAT checkpoint.
This release is intended for early testing, inference compatibility testing, independent evaluation, and community feedback.
Graphite Alpha is the larger member of the current Alpha pair. A reduced-footprint sibling, Basalt Alpha 24B-A3.8B, is also available for environments where lower memory usage and compute requirements are preferred.
Alpha release: model behavior, compatibility, formatting, architecture details, and weights may change in future revisions.
Model Overview
| Property | Value |
|---|---|
| Model | Graphite Alpha 26B-A4B |
| Architecture | Sparse Mixture-of-Experts |
| Total parameters | 26B-class |
| Active parameters | ~4B |
| Layers | 30 |
| Experts | 128 |
| Routed experts | Top-8 |
| Context length | 262,144 tokens |
| Sliding attention window | 1,024 tokens |
| Weight format | W4A16 |
| Quantization | Quantization-Aware Training |
| Weight precision | INT4 |
| Activation precision | BF16 |
| Quantization group size | 32 |
| Quantization format | compressed-tensors |
| Release status | Alpha / Experimental |
Parameter counts reported by individual libraries or model hosting platforms may differ slightly depending on whether embeddings, auxiliary parameters, routing components, and other tensors are included in the total.
The active parameter count is approximate and refers to the portion of the model typically involved in computation for a token under its sparse routing configuration.
Quantization
Graphite Alpha is released as a Quantization-Aware Trained (QAT) model rather than a conventional post-training quantized checkpoint.
The weights use symmetric 4-bit group quantization with a group size of 32 through the compressed-tensors format, while activations remain at higher precision.
Quantization is incorporated into training so that the model can adapt to low-bit weight representation rather than being converted only after higher-precision training has completed.
The goal is to reduce model size and inference memory requirements while preserving the behavior and capability of the higher-precision model as closely as practical under low-bit inference.
Context
Graphite Alpha supports a maximum configured context length of:
262,144 tokens
The architecture also uses a local sliding attention window of:
1,024 tokens
Very-long-context behavior remains experimental.
Actual usable context length and quality may vary depending on:
- inference backend
- available accelerator memory
- KV-cache precision and implementation
- batching configuration
- prompt structure
- generation length
- runtime-specific attention implementations
A backend successfully loading the configured context length does not necessarily imply identical model quality across the entire context window.
Recommended Generation Settings
The default generation configuration provides a reasonable starting point:
temperature = 1.0
top_p = 0.95
top_k = 64
These values are not strict requirements.
Different workloads may benefit from different sampling configurations.
For deterministic, reproducibility-focused, or benchmark-oriented evaluation, use the generation settings required by the benchmark rather than these defaults.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "0x795E2795/Graphite-Alpha-26B-A4B-it-qat-w4a16-ct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
)
messages = [
{
"role": "user",
"content": "Explain quantization-aware training in simple terms."
}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(
inputs,
max_new_tokens=512,
temperature=1.0,
top_p=0.95,
top_k=64,
do_sample=True,
)
response = tokenizer.decode(
outputs[0][inputs.shape[-1]:],
skip_special_tokens=True,
)
print(response)
Inference support for low-bit compressed-tensors checkpoints depends on the backend and software version being used.
Some runtimes may load the checkpoint while internally using a different execution path than native W4A16 kernels. Performance and memory usage should therefore be measured on the target runtime rather than inferred solely from the stored weight format.
Benchmarks
Benchmark evaluation is currently in progress.
Results will be published after the evaluation suite has been completed and the measurements have been verified.
The evaluation is intended to cover areas including:
- general knowledge
- reasoning
- mathematics
- coding
- instruction following
- quantization quality retention
- long-context behavior
- behavioral consistency
- inference efficiency
Preliminary or incomplete benchmark numbers are intentionally not included in this model card.
Alpha Testing
This release is primarily intended to collect real-world observations from different inference environments and workloads.
Useful feedback includes:
- inference backend and version
- GPU / accelerator
- generation configuration
- context length
- KV-cache configuration
- memory usage
- inference speed
- output quality
- instruction-following behavior
- long-context behavior
- reproducible compatibility issues
- comparisons against other quantization formats
Independent benchmark results are also welcome.
When comparing results, please include enough information to distinguish model behavior from differences introduced by runtime, prompt formatting, quantization kernels, or sampling configuration.
Known Limitations
Graphite Alpha is an experimental checkpoint.
Current limitations may include:
- backend-specific compatibility issues
- incomplete support for native W4A16 execution in some inference frameworks
- behavior changes introduced by low-bit quantization
- sensitivity to sampling configuration
- inconsistent behavior at extremely long context lengths
- runtime-specific differences in memory consumption and throughput
- higher memory requirements than aggressively packed GGUF quantizations
- differences between inference implementations
Outputs should not be assumed to be bit-for-bit or behaviorally identical across different runtimes.
Performance measured on one backend may also not generalize directly to another backend.
Intended Use
Graphite Alpha is intended for:
- research
- experimentation
- model evaluation
- local inference
- quantization research
- software development
- inference-engine testing
- runtime compatibility testing
- sparse-model experimentation
As an Alpha release, it should be evaluated carefully before use in reliability-sensitive applications.
License
This model follows the Gemma 4 license and usage terms.
Users are responsible for reviewing and complying with the applicable Gemma license terms and usage restrictions when downloading, modifying, redistributing, or using this model.
Disclaimer
Graphite Alpha is provided as an experimental model release.
Generated outputs may be inaccurate, inconsistent, biased, or otherwise unsuitable for a particular use case.
Users should independently verify important outputs and evaluate the model for their intended application.
No guarantee is made regarding compatibility with any particular inference backend, accelerator, quantization kernel, or software version.
Feedback
Bug reports, compatibility results, benchmark measurements, and other technical feedback are welcome.
When reporting results, please include enough information to reproduce the test where possible, including the model revision, runtime, hardware, context configuration, and generation settings.