Clyx 0.3 — 635.51M BASE
Clyx 0.3 BASE is a 635.51M-parameter decoder-only causal language model trained from scratch. It is a research model for text continuation, language-model experiments, and further training.
This release is a BASE checkpoint. It is not an instruction-tuned chatbot: prompts are continuation contexts, and the model is not expected to answer questions, obey formatting requests, or use tools reliably.
No external pretrained weights were used.
Model summary
| Property | Value |
|---|---|
| Parameters | 635,512,320 |
| Architecture | Decoder-only Transformer |
| Hidden size | 1,536 |
| Transformer layers | 24 |
| Query attention heads | 24 |
| Key/value attention heads | 4 |
| Attention | Grouped-query attention (GQA) |
| Intermediate size | 4,096 |
| Normalization | RMSNorm |
| Activation | SwiGLU |
| Positional encoding | RoPE |
| RoPE base | 10,000 |
| Maximum context | 2,048 tokens |
| Vocabulary | 32,768 tokens |
| Tokenizer | ByteLevel BPE |
| Embeddings | Tied input/output embeddings |
| Export format | Safetensors |
| Export precision | FP32 |
Training
The model was trained from scratch with a custom PyTorch implementation. The prepared corpus contained approximately 2.12B training tokens and approximately 20.8M validation tokens.
The training run targeted approximately 2B presented training tokens. The public export is based on the best validation checkpoint saved during that run.
| Setting | Value |
|---|---|
| Training hardware | NVIDIA A100 SXM4 80GB |
| Training precision | BF16 mixed precision |
| Optimizer | Fused AdamW |
| Learning-rate schedule | Cosine decay with warmup |
| Initial learning rate | 3e-4 |
| Minimum learning rate | 3e-5 |
| Gradient clipping | 1.0 |
| Optimization sequence length | 1,024 tokens |
| Model context capacity | 2,048 tokens |
| Target training budget | 2,000,011,264 presented tokens |
Training curve
The chart shows the recorded evaluation checkpoints from the BASE training run. The best validation loss was 2.7136 at step 116,000; the last recorded checkpoint was step 118,000 with validation loss 2.7154. Accuracy lines connect recorded observations for readability; no intermediate accuracy values were inferred. Accuracy is next-token accuracy, not an instruction-following or task benchmark score.
Training data
The corpus contained a mixture of:
- Russian text;
- English text;
- Python code;
- C and C++ code.
The private prepared corpus is not included in this repository. The model repository contains only the resulting inference weights, configuration, tokenizer, documentation, and export metadata.
The source datasets have their own licenses and terms. The Apache-2.0 license on this repository does not replace the obligations attached to the original training sources. Review source licenses before commercial redistribution or using the model in a product.
Intended use
Suitable uses include:
- multilingual text continuation experiments;
- studying a custom Transformer implementation;
- evaluating small from-scratch language models;
- further pretraining on a compatible corpus;
- instruction tuning or domain adaptation performed by the user.
This checkpoint is not intended to be used as a factual database, autonomous agent, safety-critical system, or production conversational assistant without additional training and evaluation.
Limitations
The model is not instruction-tuned, aligned, or safety-trained. It can produce incomplete, repetitive, grammatically incorrect, factually incorrect, or incoherent text. Performance varies substantially by language, domain, prompt, and sampling settings.
No MMLU, GSM8K, ARC, HumanEval, or other benchmark scores are claimed for this release. A lower language-model loss does not by itself establish instruction-following or reasoning ability.
Inference with ClyxBox
Install the separate inference runtime from PyPI:
python -m pip install clyxbox
Load the model directly from the Hugging Face repository:
from clyxbox import ClyxPipeline
pipe = ClyxPipeline.from_pretrained(
"syntropic-clx/Clyx_0.3-635.51M-BASE",
device="cuda",
)
text = pipe(
"Once upon a time in a dark forest",
max_new_tokens=128,
temperature=0,
top_p=0.95,
top_k=50,
repetition_penalty=1.4,
stream=True,
)
print(text)
When dtype is omitted, ClyxBox uses native BF16 on supported CUDA GPUs and FP32 otherwise (including NVIDIA T4). The default continuation settings are temperature=0.65, top_p=0.90, top_k=50, and repetition_penalty=1.12. Sampling is stochastic; different runs can produce different continuations. These settings are a starting point, not a guarantee of quality.
Because this is a BASE model, a prompt such as 2 + 2 = is treated as text to continue. It is not a reliable request for the answer 4. Instruction-following requires a separate SFT or instruction-tuning stage.
Sharded weights
The 2.54 GB FP32 export is intentionally split into three Safetensors files:
model-00001-of-00003.safetensors;model-00002-of-00003.safetensors;model-00003-of-00003.safetensors.
model.safetensors.index.json maps every tensor to its shard. Keep the index and all three shards in the same repository. ClyxBox reads the index automatically and loads the shards one at a time; do not rename or delete individual parts.
Repository files
model-00001-of-00003.safetensors— first weight shard;model-00002-of-00003.safetensors— second weight shard;model-00003-of-00003.safetensors— third weight shard;model.safetensors.index.json— shard-to-tensor index;model_config.json— Clyx architecture configuration;config.json— export/runtime metadata;tokenizer.jsonand tokenizer metadata — matching tokenizer;export_manifest.json— file sizes, hashes, and export information;LICENSE— Apache License 2.0;README.md— this model card.
Training checkpoints, optimizer state, private corpus files, logs, and SFT checkpoints are intentionally excluded.
Related resources
- Inference runtime: clyxbox on PyPI
- Project organization: syntropic-clx
License
Apache License 2.0.