Axion 1B FineWeb-Edu — Step 200K
This is an early base-model checkpoint from the Axion three-stage pipeline. It has 1,275,168,768 parameters and was pretrained on FineWeb-Edu with the Axion 16K SentencePiece tokenizer.
Checkpoint
- Optimizer step: 200,000
- Training tokens seen: 13,107,200,000
- Validation cross-entropy: 2.457392
- Validation perplexity: 11.67432
- Context length: 32,768 tokens
- Weight format: BF16 SafeTensors
This checkpoint is not instruction-tuned or chat-tuned. It may generate incorrect, repetitive, biased, or otherwise unreliable text.
Transformers
Custom model code is included because Axion is not a stock Llama architecture.
Review the repository code before enabling trust_remote_code.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "YOUR_NAMESPACE/YOUR_REPOSITORY"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype="bfloat16",
device_map="auto",
)
inputs = tokenizer("The future of science is", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))
vLLM
Axion uses vLLM's Transformers modeling backend:
vllm serve YOUR_NAMESPACE/YOUR_REPOSITORY \
--trust-remote-code \
--model-impl transformers \
--dtype bfloat16 \
--max-model-len 32768
For a 12GB GPU, begin with --max-model-len 8192 to leave sufficient room for
the KV cache. A 24GB GPU can use the full context more comfortably.
Upload
From the Axion repository:
hf repos create YOUR_NAMESPACE/YOUR_REPOSITORY --private --exist-ok
hf upload-large-folder YOUR_NAMESPACE/YOUR_REPOSITORY hf_export/macho-1b-finewebedu-20b-step200000
No model license is asserted by this export. The repository owner should select and document a license before making the model public.