Flash-Archer-150M
Flash-Archer-150M is a compact Llama-compatible causal language model trained from scratch on roughly 975M tokens from FineWeb-Edu.
Why This Model Exists
Flash-Archer-150M is intended as a small, easy-to-load baseline for experimentation with custom pretraining, tokenizer behavior, low-cost inference, and continued fine-tuning. It is not instruction-tuned and should be treated as a base language model rather than a chat assistant.
The main points of interest are:
- 150M parameter Llama-style decoder-only architecture.
- Custom 16k byte-level BPE tokenizer.
- Standard Hugging Face Transformers format with
safetensors. - Small enough to run on CPU for testing, with GPU recommended for faster generation or fine-tuning.
Model Details
| Field | Value |
|---|---|
| Architecture | Llama-compatible decoder-only causal LM |
| Transformers class | LlamaForCausalLM |
| Parameters | 150,335,232 |
| Layers | 18 |
| Hidden size | 768 |
| Attention heads | 12 |
| Key/value heads | 12 |
| MLP intermediate size | 2,304 |
| Activation | SiLU |
| Positional embeddings | RoPE, theta 10,000 |
| Normalization | RMSNorm, epsilon 1e-5 |
| Vocabulary size | 16,000 |
| Tokenizer | Custom byte-level BPE, PreTrainedTokenizerFast |
| Special tokens | <pad>, <unk>, <bos>, <eos> |
| Context length | 1,024 tokens |
| Checkpoint format | Transformers + model.safetensors |
| Weight dtype in uploaded checkpoint | float32 |
| License | Apache-2.0 for model weights |
Training Data
Flash-Archer-150M was trained on:
- Dataset:
HuggingFaceFW/fineweb-edu - Config:
sample-10BT - Dataset license: ODC-BY
- Documents consumed: 877,735
- Tokens seen: 974,520,320
- Validation holdout: 2,000 documents
- Tokenizer training documents: 200,000
FineWeb-Edu is an English educational web dataset derived from Common Crawl. The model card attributes the dataset here because the training data is released under ODC-BY.
Training Methodology
The model was trained from scratch as a next-token prediction causal language model.
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW |
| Peak learning rate | 3e-4 |
| Final learning rate | 3e-5 |
| Schedule | Cosine decay with 2% warmup |
| Beta1 / Beta2 | 0.9 / 0.95 |
| Adam epsilon | 1e-8 |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Micro batch size | 1 |
| Gradient accumulation | 128 |
| Training precision | FP16 mixed precision |
| Max sequence length | 1,024 |
| Final checkpoint step | 7,435 |
Validation
These are validation-log results from the training run, not standardized benchmark results. The run used a small validation setting (val_batches=2), so the numbers should be interpreted as rough training diagnostics.
| Checkpoint | Step | Tokens seen | Validation loss | Perplexity |
|---|---|---|---|---|
| Best logged validation | 4,200 | 550,502,400 | 3.0669 | 21.48 |
| Final logged validation | 7,400 | 969,932,800 | 3.3003 | 27.12 |
No public benchmark scores are claimed.
Hardware
Training metadata indicates the run was performed on Kaggle with one NVIDIA Tesla T4 GPU with about 15.6 GB VRAM. The configured maximum training time was 7 hours.
Intended Uses
This model is best suited for:
- Educational experiments with small pretrained language models.
- Continued pretraining or supervised fine-tuning.
- Tokenizer and architecture experiments.
- Lightweight local generation tests.
- Baseline comparisons against other small causal LMs.
Limitations
- This is a base model, not an instruction-following chat model.
- It may repeat itself, produce low-quality generations, or fail to follow prompts.
- It has not been evaluated on safety, factuality, coding, reasoning, or domain benchmarks.
- It was trained on web text and may reproduce biases, errors, or unsafe content present in that data.
- The context length is limited to 1,024 tokens.
- Validation metrics are noisy because only a small number of validation batches were logged.
Inference
Install dependencies:
pip install torch transformers safetensors
Load the model:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Norman89107/Flash-Archer-150M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()
prompt = "In simple terms, machine learning is"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
output_ids = model.generate(
**inputs,
max_new_tokens=64,
do_sample=True,
temperature=0.8,
top_p=0.95,
top_k=50,
repetition_penalty=1.1,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
For GPU inference:
model = AutoModelForCausalLM.from_pretrained(
"Norman89107/Flash-Archer-150M",
torch_dtype=torch.float16,
device_map="auto",
)
Files
config.json: Llama-compatible model configuration.model.safetensors: Model weights.tokenizer.json: Custom byte-level BPE tokenizer.tokenizer_config.json: Tokenizer metadata and max length.special_tokens_map.json: Special token definitions.generation_config.json: Default sampling settings.inference.py: Minimal inference script.
Source Code
No public source repository link was provided for this release. The uploaded checkpoint is in standard Transformers format and can be loaded directly with AutoModelForCausalLM.
License
The model weights are released under Apache-2.0. The training data used FineWeb-Edu, which is released under ODC-BY; see the FineWeb-Edu dataset card for dataset terms and attribution details.