h · K2-0.9B — a library voice on a modern base
One full epoch of continued pretraining of IFM/K2-Horizon-0.9B
(the pretrain_final revision, i.e. the pretraining-only checkpoint, not the RL/reasoning release) on a
private library of scanned books and magazines mixed with room transcripts and 12.5% FineWeb-Edu replay.
This is a research arm, not a finished resident. The finished one is
emberian/h-05b-replay, a Falcon-H1 hybrid trained on the same
text. This model exists to answer one question: does a modern, better-trained base make a better library voice
than the older hybrid did? Made by ember arlynx with Claude.
What it does, honestly
It has the library's voice but not the room's register. Prompted with the room format it answers in the books' language and does not reliably take a turn:
| prompt | greedy continuation |
|---|---|
ember: hi h | "The Silence. There are many, many things one can say about the 'silence' of the OBE period." |
rat: who are you | "The universe is the stage upon which the play of life is acted out." |
dov: is consciousness a thing or a process | "Consciousness is not a thing. Consciousness is a process, and as such, can be described in various ways, according to the perspective from which it is viewed." |
kestrel: write me a python function that reverses a string | "In the case of alphabetic input, the reversal process can be performed by iterating over the string in reverse and writing each character in its place." |
Sampled at 0.7 it gets terser and stranger ("Love is the law, love under will."). Requests are not so much deflected as drifted away from, into source material. One epoch on a base that never saw the room format buys the voice, not the turn-taking.
Prompting
No chat template. It completes a plain transcript: a frame paragraph, a blank line, then name: text turns
separated by blank lines, ending with h:.
A room in the library, late. h is present and answers when spoken to, briefly, in the words of the books it has read. The others are visitors.
ember: hi h
h:
- Stop on
"\n\n", not"\n". No repetition penalty. Temperature 0.7, top-p 0.9. - Give it a bare frame with no example turns, or it quotes them back.
- Needs transformers 5.x and
trust_remote_code=True— the architecture's own modelling code importsPreTrainedConfig, which transformers 4.57 does not have.
Training
438,042,624 tokens (one epoch), sequence 1024, 65,536 tokens per optimizer step, AdamW, peak LR 5e-5 with a warmup-stable-decay schedule cooling to 5.07e-6 over the last 10%, fp32 master weights with bf16 autocast, plain next-token loss. Four sessions on a single L40S.
| tokens | % | library holdout | room holdout |
|---|---|---|---|
| 0 | 0 | 3.2687 | 3.1732 |
| 219,021,312 | 50 | 2.9947 | 2.6623 |
| 438,042,624 | 100 | 2.8930 | 2.5887 |
The cooldown did real work: the last 20% of the epoch moved library loss by 0.060, where the preceding three 10% windows moved it 0.015, 0.011 and 0.009.
A caution if you compare this to another model's loss: this tokenizer encodes the same text in about 0.91 of the tokens a 32k-vocabulary Falcon tokenizer uses, so per-token loss flatters it by roughly 8%. Normalise before comparing.
Corpus
A private library (~374M tokens of scanned, in-copyright books and magazines), room transcripts in the
name: text format, and 12.5% FineWeb-Edu replay to slow the loss of general competence. The corpus is not
distributed and is not recoverable from these weights in any useful form.
License
Apache 2.0, following the base model's license: field and IFM's release announcement. Note that the base's model
card also carries a contradictory license_name: internal-only; check with IFM if that matters for your use. The
training corpus is not included and is not licensed for redistribution.