Gene-R1-1B Bio FOCUS Checkpoint
This is a workshop checkpoint initialized from ncbi/Gene-R1-1B after replacing
non-Latin tokenizer slots with HGNC protein-coding gene symbols and running
FOCUS embedding initialization.
Tokenizer Change
- Base model:
ncbi/Gene-R1-1B - Vocabulary size: 128256
- Inserted gene tokens: 19196
- Gene insertion mode: added special tokens
- Expected behavior: each inserted gene symbol tokenizes to exactly one token id
FOCUS Run
- Training split: GO SFT train split
- Train rows: 2400
- Branches: BP 800, MF 800, CC 800
- FOCUS training text rows:
- 2400 full SFT text rows
- 2400 train gene-set rows
- 150 gene-vocabulary chunk rows
- 4950 total rows
- fastText epochs: 3
- fastText dimension: 128
- fastText min_count: 1
- processes: 64
The output LM head was left unchanged because the downstream benchmark output is ordinary English biological process text; the modified inputs are gene symbols.
Intended Use
This checkpoint is intended for tokenizer-transfer workshop experiments before subsequent MATT transfer/fine-tuning. It is not a final benchmarked model.