Basalt Alpha 24B-A3.8B
Basalt Alpha is an experimental 24B-class sparse Mixture-of-Experts language model with approximately 3.8B active parameters per token, released as a W4A16 QAT checkpoint.
It is designed as a reduced-footprint sibling to Graphite Alpha 26B-A4B, targeting lower model size and compute requirements while retaining the same general model family and deployment characteristics.
This release is intended for early testing, inference compatibility testing, independent evaluation, efficiency testing, and community feedback.
Alpha release: model behavior, compatibility, formatting, architecture details, and weights may change in future revisions.
Model Overview
| Property | Value |
|---|---|
| Model | Basalt Alpha 24B-A3.8B |
| Architecture | Sparse Mixture-of-Experts |
| Total parameters | 24B-class |
| Active parameters | ~3.8B |
| Context length | 262,144 tokens |
| Sliding attention window | 1,024 tokens |
| Weight format | W4A16 |
| Quantization | Quantization-Aware Training |
| Weight precision | INT4 |
| Activation precision | BF16 |
| Quantization group size | 32 |
| Quantization format | compressed-tensors |
| Release status | Alpha / Experimental |
Parameter counts reported by individual libraries or model hosting platforms may differ slightly depending on whether embeddings, auxiliary parameters, routing components, and other tensors are included in the total.
The active parameter count is approximate and refers to the portion of the model typically involved in computation for a token under its sparse routing configuration.
Basalt Alpha reduces both total and active parameter counts relative to Graphite Alpha:
| Model | Total Parameters | Active Parameters |
|---|---|---|
| Graphite Alpha | 26B-class | ~4B |
| Basalt Alpha | 24B-class | ~3.8B |
The purpose of this reduction is to provide a lighter deployment target rather than to create a fundamentally separate model family.
Quantization
Basalt Alpha is released as a Quantization-Aware Trained (QAT) model rather than a conventional post-training quantized checkpoint.
The weights use symmetric 4-bit group quantization with a group size of 32 through the compressed-tensors format, while activations remain at higher precision.
Quantization is incorporated into training so that the model can adapt to low-bit weight representation rather than being converted only after higher-precision training has completed.
Combined with its reduced parameter footprint, this makes Basalt Alpha particularly suited for experimentation on systems where model memory, bandwidth, or compute availability is more constrained.
The goal is to retain as much of the larger model's behavior as practical while reducing inference cost.
Context
Basalt Alpha supports a maximum configured context length of:
262,144 tokens
The architecture also uses a local sliding attention window of:
1,024 tokens
Very-long-context behavior remains experimental.
Actual usable context length and quality may vary depending on:
- inference backend
- available accelerator memory
- KV-cache precision and implementation
- batching configuration
- prompt structure
- generation length
- runtime-specific attention implementations
Basalt's smaller parameter footprint may provide additional memory headroom for KV cache or larger workloads on memory-constrained systems, but the practical benefit depends heavily on the inference backend and cache configuration.
Recommended Generation Settings
The default generation configuration provides a reasonable starting point:
temperature = 1.0
top_p = 0.95
top_k = 64
These values are not strict requirements.
Different workloads may benefit from different sampling configurations.
For deterministic, reproducibility-focused, or benchmark-oriented evaluation, use the generation settings required by the benchmark rather than these defaults.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "0x795E2795/Basalt-Alpha-24B-A3.8B-it-qat-w4a16-ct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
)
messages = [
{
"role": "user",
"content": "Explain quantization-aware training in simple terms."
}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(
inputs,
max_new_tokens=512,
temperature=1.0,
top_p=0.95,
top_k=64,
do_sample=True,
)
response = tokenizer.decode(
outputs[0][inputs.shape[-1]:],
skip_special_tokens=True,
)
print(response)
Inference support for low-bit compressed-tensors checkpoints depends on the backend and software version being used.
Some runtimes may load the checkpoint while internally using a different execution path than native W4A16 kernels. Performance and memory usage should therefore be measured on the target runtime rather than inferred solely from the stored weight format.
Graphite and Basalt
Graphite Alpha and Basalt Alpha are related Alpha releases targeting different deployment points.
Graphite Alpha 26B-A4B prioritizes the larger model configuration, with approximately 26B total parameters and 4B active parameters per token.
Basalt Alpha 24B-A3.8B reduces the model footprint to approximately 24B total parameters and 3.8B active parameters per token.
Basalt is intended for situations where lower storage, memory bandwidth, or computational requirements are valuable while retaining a model in the same general capability and architecture family.
The two checkpoints should still be evaluated independently. Reduced parameter count does not imply that performance differences will be uniform across tasks.
Benchmarks
Benchmark evaluation is currently in progress.
Results will be published after the evaluation suite has been completed and the measurements have been verified.
The evaluation is intended to cover areas including:
- general knowledge
- reasoning
- mathematics
- coding
- instruction following
- quantization quality retention
- long-context behavior
- behavioral consistency
- inference efficiency
- memory efficiency
- comparison with Graphite Alpha
Preliminary or incomplete benchmark numbers are intentionally not included in this model card.
For Graphite/Basalt comparisons, identical prompts, evaluation settings, runtime versions, and sampling configurations should be used wherever possible.
Alpha Testing
This release is primarily intended to collect real-world observations from different inference environments and workloads.
For Basalt Alpha, efficiency-oriented testing is especially useful.
Useful feedback includes:
- inference backend and version
- GPU / accelerator
- generation configuration
- context length
- KV-cache configuration
- memory usage
- inference speed
- output quality
- instruction-following behavior
- long-context behavior
- reproducible compatibility issues
- comparisons against Graphite Alpha
- comparisons against other quantization formats
Independent benchmark results are also welcome.
When comparing Basalt with Graphite, please include enough information to distinguish architecture-level differences from differences introduced by runtime, prompt formatting, quantization kernels, or sampling configuration.
Known Limitations
Basalt Alpha is an experimental checkpoint.
Current limitations may include:
- backend-specific compatibility issues
- incomplete support for native W4A16 execution in some inference frameworks
- behavior changes introduced by low-bit quantization
- capability differences resulting from the reduced model footprint
- sensitivity to sampling configuration
- inconsistent behavior at extremely long context lengths
- runtime-specific differences in memory consumption and throughput
- higher memory requirements than aggressively packed GGUF quantizations
- differences between inference implementations
Basalt should not be assumed to reproduce Graphite outputs exactly.
Differences between the two models may vary significantly depending on the task.
Outputs should also not be assumed to be bit-for-bit or behaviorally identical across different runtimes.
Intended Use
Basalt Alpha is intended for:
- research
- experimentation
- model evaluation
- local inference
- memory-constrained inference
- quantization research
- software development
- inference-engine testing
- runtime compatibility testing
- sparse-model experimentation
- efficiency evaluation
As an Alpha release, it should be evaluated carefully before use in reliability-sensitive applications.
License
This model follows the Gemma 4 license and usage terms.
Users are responsible for reviewing and complying with the applicable Gemma license terms and usage restrictions when downloading, modifying, redistributing, or using this model.
Disclaimer
Basalt Alpha is provided as an experimental model release.
Generated outputs may be inaccurate, inconsistent, biased, or otherwise unsuitable for a particular use case.
Users should independently verify important outputs and evaluate the model for their intended application.
No guarantee is made regarding compatibility with any particular inference backend, accelerator, quantization kernel, or software version.
Feedback
Bug reports, compatibility results, benchmark measurements, efficiency measurements, and other technical feedback are welcome.
When reporting results, please include enough information to reproduce the test where possible, including the model revision, runtime, hardware, context configuration, and generation settings.