Morpheus-FIM (Mamba-2) — Basque Fill-in-the-Middle Autocomplete
Continued pre-training of itzune/morpheus (the 91M AR-only Mamba-2 model, step 74K) for Fill-in-the-Middle (FIM) completion — the model can predict text at the cursor, not just at the end of a buffer.
This is the base model for the GGUF at itzune/morpheus-gguf (v3_fim.Q5_K_M.gguf).
Model Details
- Architecture: Mamba-2 (State Space Model), 24 layers, d_model 768
- Parameters: 91M
- Base model: itzune/morpheus (step 74K, AR-only, PPL 7.13)
- Checkpoint: Phase 6 v3
best.pt(step 3500 of CPT) - AR perplexity: 7.5 | FIM perplexity: 7.9
- Vocabulary: 4,016 (4,000 original Unigram pieces +
<PRE><SUF><MID><EOT>+ 12 padding rows) - Trained without BOS token (
add_bos_token=false)
Continued Pre-Training Recipe
| Parameter | Value |
|---|---|
| Base checkpoint | step 74K AR (itzune/morpheus), embeddings resized 4000 → 4016 |
| Token budget | 500M tokens |
| FIM/AR ratio | 70/30 |
<EOT> loss weight | 5× (per-class cross-entropy weight on token id 4003) |
| Splitting | Token-level (BigCode/StarCoder), 20% at linguistic boundaries |
| Loss masking | None ("FIM-for-free" — loss on all tokens) |
| Packing | Greedy whole-example packing into 1025-token windows |
| Learning rate | 1.0e-3, cosine decay, ~3,815 steps |
| Tokenizer | basque_unigram_fim.model (original 4000 + 4 FIM tokens) |
The 70/30 ratio + 5× <EOT> weight directly target the FIM stop-token reliability problem: the model must learn not just what to generate but when to stop, and the <EOT> signal is otherwise too sparse (one token per example) within a 500M-token budget for the model to reliably emit it.
FIM Token Format
Code Llama-style FIM tokens (Bavarian et al., 2022; Roziere et al., 2023):
| Token | ID | Purpose |
|---|---|---|
<PRE> | 4000 | Marks start of prefix |
<SUF> | 4001 | Marks start of suffix |
<MID> | 4002 | Marks start of generation (infill) |
<EOT> | 4003 | End-of-infill (stop token) |
| 4004–4015 | — | Padding (kernel alignment, unused) |
To do a fill-in-the-middle completion, structure the prompt as:
<PRE>{prefix}<SUF>{suffix}<MID>
The model generates the infill and emits <EOT> when done.
Evaluation Results
FIM eval on 147 held-out examples (token-level splits, 20% at linguistic boundaries):
| Metric | Result |
|---|---|
<EOT> emission rate | 88.4% |
| Keystrokes saved | −5.9% |
| Exact-match rate | 6.8% |
| Avg char accuracy | 32.3% |
| Avg generation length (ref=45.0) | 40.3 |
| Prefix truncation (overall) | 1.4% |
| └ long-bucket truncation | 2.3% (< 15% threshold) |
| AR valid PPL | 7.5 |
| FIM valid PPL | 7.9 |
The 5× <EOT> loss weighting resolves the over-generation failure mode: <EOT> emission reaches 88.4% and generation length (40.3) sits near the 45.0-char reference, yielding near-break-even keystrokes saved (−5.9%). The feared premature-truncation failure mode — the dual risk of over-weighting the stop token — did not materialize (1.4% overall, 2.3% long-bucket, far below the 15% threshold). AR perplexity remained stable (7.5 vs. 7.13 AR-only base), confirming the 70/30 FIM ratio did not trade away AR capability.
Usage
With the Morpheus demo server (recommended)
The Morpheus demo includes a FastAPI proxy that handles FIM templating, token-ID encoding, and an OpenAI-compatible API:
cd demo
MORPHEUS_MODEL=v3_fim.Q5_K_M.gguf docker compose -f docker-compose.yml -f docker-compose.local.yml up -d --build
# Open http://localhost:9090/editor.html
Direct with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("itzune/morpheus-fim")
tokenizer = AutoTokenizer.from_pretrained("itzune/morpheus-fim")
# FIM prompt
prompt = "<PRE>Kaixo, <SUF> moduz?<MID>"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=20, temperature=0.2, top_k=5)
print(tokenizer.decode(output[0], skip_special_tokens=False))
⚠️ Critical: token-ID prompts for llama.cpp
When deploying via llama.cpp/llama-server, encode prompts with the sentencepiece library using tokenizer.model and send token IDs (not strings) to the /completion endpoint. This avoids the BOS auto-prepend and tokenizer-divergence issues documented on the base model card. The demo proxy handles this automatically.
Decoding Parameters (recommended)
| Parameter | Value | Rationale |
|---|---|---|
temperature | 0.2 | Low-but-nonzero: recovers rank-2 correct tokens greedy misses |
top_k | 5 | Small nucleus; 5 correct answers sit at rank 2 in top-5 |
repeat_penalty (FIM) | 1.0 | FIM legitimately reuses context words |
stop (FIM) | ["<EOT>", "\n\n"] | Model-emitted stop + paragraph-boundary fallback |
Intended Use
Desktop text-editor ghost-text autocompletion for Basque prose. The Mamba-2 architecture's O(1) decode cost makes it well-suited to long editing sessions where per-token latency matters more than parallelism.
Not intended for: instruction following, chat, translation, or factual QA. This is a narrow autocomplete model.
License
Apache-2.0.
References
- Bavarian, M., et al. (2022). Efficient Training of Language Models to Fill in the Middle. arXiv:2207.14255.
- Roziere, B., et al. (2023). Code Llama: Open Foundation Models for Code. arXiv:2308.12950.
- Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms through Structured State Space Duality. arXiv:2405.21060.