PASITA v1 — plain text to Markdown (faithful)
A decoder-only language model trained 100% from scratch (no base model), specialized in a single task: converting plain text into valid Markdown while preserving information.
Compiler behavior, not chatbot behavior: it adds structure, it does not invent content.
Architecture (lab notes)
| Parameter | Value |
|---|---|
| Class | LlamaForCausalLM (decoder-only, dense, no MoE) |
| Parameters | 88,099,584 (~88M) in bfloat16 |
| Layers / hidden / FFN | 12 / 768 / 2048 (SwiGLU) |
| Attention | GQA 12Q/4KV, head_dim 64, no bias |
| Positions | RoPE theta=100000, ctx 2048, RMSNorm eps=1e-5 |
| Embeddings | tied (saves ~12.6M params) |
| File | model.safetensors (176 MB, 110 tensors, sha 664665e7…) |
| Tokenizer | Custom 16k byte-level BPE, verified ByteLevel decoder (~4.0 chars/token ES/EN) |
| Special tokens | <pad> <s> </s> <unk> <think> </think> |
Training
- SFT 58M tokens x2 epochs — final loss 0.09, token accuracy 98.4%
- DPO beta=0.1 — preference margin 4.2
- GRPO 550+150 steps, G=4, verifiable rewards (format + numeric fidelity + anti-overformatting)
- Data: 80M human markdown-derived tokens (ES/EN Wikipedia, StackExchange, WikiHow) scaled to 625M in v4 corpus
Benchmark (held-out n=1000, greedy)
| Global | GFM 0.956 - faith 0.927 - sem 0.888 - tables 1.0 |
|---|---|
| code / ocr / docs / math / tables / html | 0.93 - 1.00 |
| control (strict instructions) | 0.19 (known limitation) |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("OpceanAI/PASITA")
model = AutoModelForCausalLM.from_pretrained("OpceanAI/PASITA", dtype="auto")
prompt = "CONVIERTE A MARKDOWN:\n" + text + "\n\n### Markdown:\n"
ids = tok(prompt, return_tensors="pt", truncation=True, max_length=1024)
out = model.generate(**ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
Valid regime: medium/long documents (OCR output, pasted HTML, meeting notes, tutorials). Fragile on 1-3 line inputs.
Limitations
May truncate digits, drop secondary data, emit echo H1s, or continue past completion. Recommended: adaptive max_new_tokens + beam search + fidelity rerank.