orb-human-typeahead-1b-v2.1
A 1.6B typeahead model that predicts the human's next few words in a roleplay chat — inline "ghost text" for the person typing, not a reply generator for the character. Full fine-tune of ibm-granite/granite-4.0-1b-base.
Most small LMs asked to continue a user's half-typed roleplay message produce fluent but irrelevant text. This model is trained specifically on (conversation context + partial user message → the words the user actually typed next), so its suggestions stay on-scene and in-voice.
Variants
| Path | Format | Use with |
|---|---|---|
/ | safetensors bf16, plain granite arch | transformers |
GGUF/orb-human-typeahead-1b-v2.1-Q4_0.gguf | GGUF Q4_0 (default) | llama.cpp / llama-cpp-python |
GGUF/orb-human-typeahead-1b-v2.1-Q8_0.gguf | GGUF Q8_0 | llama.cpp / llama-cpp-python |
The original fine-tune used the granitemoehybrid architecture; since this
variant is attention-only and dense, the weights are republished here as the
equivalent plain granite architecture (logit-identical, verified), which
loads everywhere without extras.
Prompt format
Plain text, no chat template. Optional character summary, a marker line, name-prefixed turns, and finally the user's draft — the model continues the draft. Cut the suggestion at the first newline.
<character summary, optional>
***Roleplay chat below***
Sylvara: *She looks down from the watchtower and sees you.*
Traveler: *I approach the encampment.*
Sylvara: *She lowers her bow as you approach the gate.* "State your business, traveler."
Traveler: *I raise both hands slowly and
A completion looks like step into the torchlight, keeping my voice low.*
Notes:
- Trained to trigger at word boundaries only (draft ends on a whole word, or on a trailing space). Mid-word completion is out of scope.
- Trained context: up to ~4 recent turns, summaries ≤400 chars, turns ≤500 chars.
Serving recipe
Greedy, short budget, stop at newline — mirrors how it was trained and evaluated:
from llama_cpp import Llama
llm = Llama("GGUF/orb-human-typeahead-1b-v2.1-Q4_0.gguf", n_ctx=1024)
out = llm.create_completion(prompt, max_tokens=12, stop=["\n"], temperature=0.0)
print(out["choices"][0]["text"])
Or with transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("chartreuse-verte/orb-human-typeahead-1b-v2.1")
model = AutoModelForCausalLM.from_pretrained("chartreuse-verte/orb-human-typeahead-1b-v2.1")
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=12, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).split("\n")[0])
Evaluation
Scored on a held-out validation set of real roleplay conversations (120 prompts, conversation-disjoint from training; greedy, 12-token budget, suggestions cut at newline). word-EM@k = fraction of prompts where the first k words of the suggestion exactly match what the user really typed next; prefix-chars = mean length of the exactly-matching leading characters.
| Metric | orb-human-typeahead-1b-v2 | this model |
|---|---|---|
| word-EM@1 | 0.417 | 0.483 |
| word-EM@2 | 0.217 | 0.367 |
| word-EM@3 | 0.142 | 0.267 |
| prefix-chars | 5.41 | 10.89 |
| completion perplexity | 7.34 | 3.56 |
v2.1 is a data-only refresh of v2 (same 1.6B granite base): stage-1 training data was made more diverse and its action distribution flattened, so suggestions stop collapsing onto a few stock gestures. Absolute numbers stay modest by design — creative roleplay is high-entropy, so even a perfect suggester can't guess most continuations.
Training
Two-stage full fine-tune of granite-4.0-1b-base: stage 1 on a large, mostly-synthetic roleplay corpus (self-chat generated to cover the on-scene, in-voice register real users type in), then stage 2 on a smaller private in-domain set in the serve-time prompt format (with stage-1 replay to limit forgetting). Over v2, stage 1 was rebuilt with more diverse scenes and a flattened action distribution, so the model stops falling back on a few stock gestures (v2 over-suggested “I lean back” / “crossing my arms”). Examples are user turns split at word boundaries; loss is on the continuation. Suggestions are single-line by construction (completions end at newline).
Limitations
- English-centric, roleplay register (asterisk actions, quoted dialogue). Out of domain for assistant chat, code, or formal prose.
- Roleplay corpora include mature themes; suggestions can reflect that. Intended as a typing aid for consenting adult users of RP chat apps.
- Not an instruction follower — it only continues drafts in the format above.
- Suggests at word boundaries; won't complete a half-typed word.