NoTokenLM-Gen-3.5-Focused

A 9M-parameter, byte-level, tokenizer-free language model -- fine-tuned specifically to fix its predecessor's biggest weakness: long-range coherence.

Part of the NoTokenLM family. If you haven't seen NoTokenLM-Gen-3.5 yet, read that card first -- this model is a direct, short, targeted continuation of it, not a new architecture.

This card was last auto-updated from checkpoint step 45,600 (session #9) on 2026-08-19.

What changed, and why

Gen-3.5's own model card was honest about its biggest weakness: "Long-range coherence (3+ sentences) degrades noticeably. [...] the model starts losing track of who the subject is." We measured this directly: ~38-42% of generations were fully coherent on a 100-prompt manual evaluation.

Focused is a continuation training run on top of the Gen-3.5 checkpoint, using a narrower, simpler-narrative dataset (TinyStories-only, after an earlier c4/math/synth mix was found to inject a "word-problem" register into unrelated prompts) chosen specifically to reinforce subject tracking and short-story structure, rather than adding new capability or new knowledge.

Result: on the same style of evaluation (200 generations, 10 held-out prompts, temperature 0.6), coherence rose to ~86%.

This is not a new model bolted onto the old one. It's the same 9M architecture (RoPE, RMSNorm, SwiGLU, byte-level vocabulary, weight tying), same weights carried forward, just pushed further in the one direction its predecessor was weakest at.

Real examples, unedited (from the 200-generation evaluation)

Subject tracking across 2-3 sentences, which was Gen-3.5's main failure point:

"She walked into the room and saw a big machine. She was curious and wanted to see what i[t was]"
"The door opened and a little girl came out to see what was inside. She saw the [...]"
"He looked at the sky and saw a big cloud. He saw the sun shining in the sky with the [...]"
"The little girl was so excited to go on an adventure. She wore her favorite [...]"
"He said to his mom, 'I have a surprise for you!' His Mom was very e[xcited]"

Note the consistent pronoun tracking (she/she, he/he, his/his) across multiple clauses -- this is exactly the failure mode Gen-3.5's card documented ("a 'she' quietly becomes a 'he' a few sentences later") and it's substantially reduced here.

We're not hiding the misses either. Out of 200 generations, roughly 1 in 7 still drifts or contradicts itself:

"The old man was a girl. She was very popular."          <- contradicts its own subject mid-sentence
"The little girl was a little boy who liked to walk..."   <- same failure mode, different prompt
"They walked into a big forest with a long direction."    <- grammatically fine, semantically empty

That failure rate (~14%) is real and we're reporting it, not the ~86% alone.

What this model is and isn't

• It is: the same base capability as Gen-3.5 (real word morphology, dialogue formatting, short-range grammar), with meaningfully better multi-sentence subject consistency.
• It is not: a chat or instruction-following model. It has not seen QA-formatted or instruction data. Prompt it with a sentence start, not a question expecting a direct answer.
• It still has no reliable arithmetic ability and no reliable factual/world knowledge. An earlier checkpoint (still trained on a c4/math/synth mix) showed partial operation-type recognition on word problems (correctly picking + or *) but consistently wrong arithmetic; the current TinyStories-only checkpoint doesn't attempt arithmetic at all anymore and treats math-shaped prompts as story openers instead. Neither is a reliable math capability.
• Style has shifted toward simpler, shorter-sentence narrative (closer to children's-story structure) compared to Gen-3.5's more 19th-century-novel register, as a side effect of the dataset used for this focused pass. If you want the more literary tone, use base Gen-3.5; if you want more reliable multi-sentence coherence, use this one.

How to run it

Recommended temperature: 0.5-0.6 (0.6 was used for the evaluation above).

Use the "Open in Colab" button in this repo's action menu (top of this page) for a zero-setup run -- no token needed.

From Python directly

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "omurberaisik/NoTokenLM-Gen-3.5-Focused", trust_remote_code=True
)
model.eval()

prompt = "She walked into the room and"
ids = torch.tensor([list(prompt.encode("utf-8"))], dtype=torch.long)
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.6, top_k=40)
text = bytes(x for x in out[0].tolist() if 0 <= x <= 255).decode("utf-8", errors="replace")
print(text)

No tokenizer object needed -- this is a byte-level model (256-value vocabulary, the raw UTF-8 bytes), so encode/decode is just str.encode("utf-8") / bytes.decode("utf-8").

Architecture details

What's next

Whether future continued training holds onto this checkpoint's coherence gains while adding new capability (broader world knowledge, more reliable reasoning) without regressing subject-tracking is an open, testable question -- we'll report the real numbers when we have them, same as here.

Additional evaluation: 1,000-prompt test — New

To get a larger-sample read on coherence than the 200-generation evaluation above, we ran the model on 1,000 unique prompts (raw checkpoint weights, no EMA available for this checkpoint), each generation manually read and labeled into one of the same three categories used elsewhere in the NoTokenLM family. Full generation settings, prompts, outputs, and per-example labels are in gen35_focused_1000_test_outputs.json in this repo -- nothing summarized here is cherry-picked from that file.

Generation settings: max_new_tokens=35, temperature=0.5, top_k=40, deterministic per-prompt seed.

At this larger sample size and shorter completion length (35 bytes vs. the 100 used in the 200-generation evaluation above), the fully-coherent rate lands at ~69%, with grammatically-fluent-but-semantically-odd completions making up most of the remainder and outright broken output staying rare (0.2%). This isn't directly comparable to the ~86% figure above -- different prompt set, different completion length, different sample size -- but it points the same direction: the TinyStories-focused continuation training holds up as a large, unfiltered sample, not just on the smaller hand-picked evaluation set.

Part of the NoTokenLM family -- small models, built and evaluated honestly.