PersonaMini-1 small
A 28.8M-parameter GPT-style roleplay & chat model, trained entirely from scratch by BytesTalk — no existing model was fine-tuned; every weight was learned from random initialization. It punches well above its weight on conversational behavior for its size, while staying honest about the hard limits of a model this small (see Limitations).
Mature content. This is an uncensored roleplay model and can produce adult/NSFW text. Intended for 18+ use only.
Capabilities
- Coherent, in-character short replies; holds a given persona
- Answers relevantly instead of dumping its identity
- Empathetic responses; acknowledges the user; short-context memory recall
- Roleplay and character chat — its core purpose
Limitations
- World knowledge is weak. It reliably knows only a small set of everyday concepts; for anything obscure it will guess. This is a hard parameter-count ceiling, not a fixable bug.
- Two-name role tracking ("I'm the character / you're the user") can still wobble.
- Occasional incoherence or invented details — reduced through training, not eliminated.
- Best for short roleplay / chat, not facts, math, code, or long documents.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("bytestalkai/PersonaMini-1-small")
model = AutoModelForCausalLM.from_pretrained("bytestalkai/PersonaMini-1-small").eval()
msgs = [{"role": "user", "content": "You are Mia, a flirty bartender. Hi Mia!"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80, do_sample=True, temperature=0.6,
top_p=0.9, top_k=40, repetition_penalty=1.2,
eos_token_id=50256, pad_token_id=50256)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
Recommended sampling: temperature≈0.5–0.6, top_p≈0.9, top_k≈40, repetition_penalty≈1.2.
Lower temperature reduces invented details.
Chat format
### USER:
{optional persona line}
{your message}
### ASSISTANT:
{reply}<|endoftext|>
An optional persona/system line goes at the top of the user turn, e.g. You are Luna, a shy witch.
Architecture
GPT-2-compatible decoder, exported to standard GPT2LMHeadModel (the safetensors here match the
original checkpoint to a max logit difference of ~2e-5):
| Parameters | ~28.8M |
| Layers | 8 |
| Hidden size | 384 |
| Heads | 6 |
| Context length | 256 |
| MLP | 2× hidden (768), GELU |
| Norm | pre-LayerNorm, no biases |
| Embeddings | tied input/output |
| Tokenizer | GPT-2 BPE (vocab 50257) |
Training pipeline
The model was built with a modern small-model recipe: pretrain → staged SFT → iterative distillation-by-repair. A larger instruction-following LLM was used only during data preparation as an automated judge and reviser — it is not shipped, embedded, or queried at inference time.
1. Pretraining
Trained from random initialization on a mixed corpus (~0.4B tokens) of web/educational text, short stories, and roleplay/dialogue data, so the base learns general English plus the target domain. Best-validation checkpointing was used as an early-stopping guard.
2. Supervised fine-tuning (staged, to avoid one common failure)
Instruction/roleplay SFT was split into stages because mixing everything into one blob made earlier versions bland and caused identity to "bleed" into unrelated answers:
- Stage A — rich SFT (no identity): roleplay + chat + Q&A only, to teach format, turn-taking, and immersion while preserving the base's richness.
- Stage B — light identity pass with replay: a small pass adds the model's self-identity and a few grounded definitions. It is mixed with a replay sample of Stage-A data to prevent catastrophic forgetting — without replay, an identity-only pass made the model answer every prompt with its self-introduction.
3. Iterative distillation by repair (the main quality driver)
For a bank of prompts (roleplay with diverse personas, chat, empathy, memory, identity, and persona-grounded perspective scenarios), the pipeline:
- generates several candidate replies from the current model,
- has the judge model pick the best/worst, score the best (1–5), and — when the best is weak — rewrite it into an exemplary short reply,
- uses those rewrites (plus the model's own high-scoring replies) as supervised targets (text-level distillation / rejection-sampling fine-tuning).
This was run for two rounds, each time using the improved model as the generator. The model measurably improved between rounds — average self-reply quality rose 2.55 → 3.25 / 5, and the fraction of replies needing a rewrite fell 89% → 62% — evidence the loop was closing the gap.
Measures taken (methodology notes)
- Checkpoint selection. On hard, free-form data, per-token validation loss barely beats the base even while behavior improves, so we save the final (fully-trained) checkpoint rather than the lowest-val one, which would otherwise return an essentially untrained model.
- Anti-forgetting replay. Every light continuation pass mixes in a replay sample of prior data, so new skills are added without erasing old ones.
- Judge design. The automated judge used a strict, priority-ordered rubric: relevance first (a reply that ignores the question — e.g. self-introducing instead of answering — is scored as a non-answer and rewritten), then perspective (the character is the assistant; never call the user by the character's name; recall the user's own stated name), no hallucination (never invent names/facts the user didn't provide), persona-detail use, and no repetition. Candidate order was randomized to remove position bias, and each chosen reply got an absolute quality score so only genuinely-good replies became training targets.
- Repetition handling. Replies that echo an earlier turn are penalized/rejected explicitly.
- Preference optimization tried and dropped. A DPO pass was implemented and tested, but on these teacher-vs-model pairs the quality gap was so large that DPO over-optimized (reward margin exploded, outputs degraded) regardless of the KL weight. Supervised distillation gave the real gains, so the released model is the distillation checkpoint, without DPO.
- Export verification. The safetensors and GGUF exports were checked to reproduce the original model's outputs before release (logit match ~2e-5).
Evaluation
Evaluation was behavioral, targeting the specific failure modes above: relevance, empathy, name-acknowledgement, in-character roleplay, identity, short-context memory, perspective/role tracking, and hallucination — compared side-by-side across successive versions.
Intended use & out-of-scope
Intended for short-form roleplay and casual chat by adults. Out of scope: factual question answering, reasoning/math/code, professional or safety-critical use, and any use by minors or that violates applicable law.
License
MIT. Use responsibly. 18+ only.
Family
- PersonaMini-1-small — 28.8M (this model)
- PersonaMini-1-medium — 63.2M
- PersonaMini-1-big — 160M