FABER-Genesis-A100-Max (13.86B architecture)
Decoder-only transformer with weight sharing + CASS state/value/halt heads.
- hidden 6144, heads 48, KV 8, head-dim 128, SwiGLU FFN 16384
- vocab 32768, tied embeddings, RMSNorm, RoPE, GQA
- 35 physical blocks -> 42 logical transformations + 1 recurrent pass
- total params: 13.860526082B (backbone + CASS)
- training recipe: BF16, AdamW8bit, activation checkpointing, microbatch=1, ctx=512
Profiles
| profile | physical | params | status |
|---|---|---|---|
| production-13.08B | 33 | 13.0804B | default for long runs (~73.55 GiB) |
| safe-13.47B | 34 | 13.4704B | squeeze with safety (~75.75 GiB) |
| max-13.86B | 35 | 13.8605B | capacity experiments (~77.94 GiB) |
| oom-14.25B | 36 | 14.2506B | OOM on single A100 AdamW8bit |
What this repo contains
config.json (max profile), modeling_faber_genesis.py (self-contained HF
model code), genesis_profiles.json, tokenizer_config.json (placeholder
config, vocab 32768 — train a real tokenizer before pretraining),
benchmark_genesis_a100.py, EVIDENCE_GENESIS_A100_MAX.md.
No trained weights are included in this upload. config.json defines a
random-init architecture. To materialize weights:
from modeling_faber_genesis import FaberGenesisConfig, FaberGenesisForCausalLM
cfg = FaberGenesisConfig() # max-13.86B
model = FaberGenesisForCausalLM(cfg) # random init, BF16 training recipe applies on A100
Ceiling benchmark (run on A100 80GB)
pip install "torch==2.6.0" bitsandbytes transformers
python benchmark_genesis_a100.py --profile max-13.86B --steps 1
python benchmark_genesis_a100.py --sweep --n-list 25,29,32,33,34,35,36,37 --steps 1
Pass = fwd+bwd+optim step completes; n=36/37 must OOM. VRAM estimates in
EVIDENCE_GENESIS_A100_MAX.md are analytic, not a substitute for this run.
Limitations
Random-init step-pass says nothing about Thai/code/reasoning quality. Quality requires pretraining curriculum + CASS + verifier + agent training and SWE/APEX evals. 14.25B+ is not a single-A100 resident AdamW8bit config.