Tyness BOE 1.2B — Pre-Alpha (Tyness Train v0.4.x)

The first model trained end-to-end with Tyness Train, an agent-guided fine-tuning harness for small LLMs on low-VRAM GPUs. This is a pre-alpha release: it proves the full pipeline (dataset → QLoRA SFT on a GTX 1060 6GB → merge → GGUF → evaluation) and documents honest findings, including negative ones.

⚠️ Status: PRE-ALPHA. The model learned the format of Spanish legal Q&A but hallucinates specific citations. Do not use it for legal information. See Evaluation & findings.

Model description

Why this model exists

Tyness Train is a harness for fine-tuning 1B–3B models on 6GB VRAM GPUs. This first release validates the whole pipeline on a deliberately hard, knowledge-dense domain:

1. Dataset: ChatML entries from Spanish consolidated legislation (BOE).
2. Training: QLoRA on Pascal, where bf16/fp16 compute is not usable — fp32 only.
3. Export: merged safetensors (bf16) + GGUF f16/Q8_0/Q4_K_M.
4. Evaluation: LLM-judged (glm5.3-flash) against held-out questions with expected answers.

Training data

Source: consolidated Spanish legislation published by the Agencia Estatal Boletín Oficial del Estado (BOE), reused per its open-data conditions, via legalize-es (markdown per norm, YAML frontmatter).

Entries: teacher-generated ChatML conversations (question / grounded answer), 1,148 training samples + 20 held-out eval items, filtered to 10 BOE articles:

Methodology:

• Seeded, diversity-aware article selection (≥40 entries, ≥4 question categories, mix of scopes).
• 2 held-out entries per article for evaluation (never seen in training).
• Each sample: system prompt (rigorous BOE jurist persona) + user question + assistant answer.
• Loss computed on assistant tokens only (completion_only_loss).
• Provenance metadata kept strictly separated — never part of training.

Training configuration

Evaluation & findings

Judged by an LLM referee (glm5.3-flash) on 20 held-out questions, verdict per item: correct / partial / incorrect, against the expected answer and the key reason.

Result: 0/20 correct, mean score 0.005. Report: pre-alpha baseline.

Findings (this is the value of the pre-alpha):

1. Format learned, facts not. Answers are fluent, structured legal Spanish with plausible citation patterns — but the model confabulates identifiers ("Ley 6/2015", "Reglamento (UE) 1169/2009" for a subsidies question). A 1.2B model trained for 3 epochs does not reliably memorize norm-specific facts from ~115 samples/article.
2. Generation truncation (256 tokens) cut some answers mid-argument; part of the failures may be "partial" rather than "incorrect". Fixed for the next iteration (longer generation budget).
3. The judge pipeline works: strict verdicts, useful per-item reasons, reproducible greedy decoding. The eval infrastructure is trustworthy even though this checkpoint is not.
4. The hardware pipeline works: 3 epochs on 6GB VRAM without OOM, ~1.5GB peak, 100% GPU utilization, bf16-free fp32 compute on Pascal.

Next iteration (in progress): 6 epochs, LoRA r=32, 16K context ceiling, longer eval generations — targeting fact memorization, not just format.

How to run

llama.cpp (recommended — all quantizations included in this repo)

# 4-bit (fits in ~700MB of RAM/VRAM)
llama-cli -hf jasonjimnz/tyness-boe-1.2b-pre-alpha:Q4_K_M -p "¿Cuál es el plazo para…?" --jinja

# or a local file + OpenAI-compatible server
llama-server -m tyness-boe-1.2b-pre-alpha-q4_k_m.gguf -ngl 99 -c 4096 --port 8080 --jinja
curl http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [{"role":"system","content":"Eres un jurista experto en el BOE. Responde con rigor y cita identificadores oficiales. Si un dato no consta, decláralo."},
               {"role":"user","content":"¿Qué es el modelo 179?"}],
  "temperature": 0
}'

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("jasonjimnz/tyness-boe-1.2b-pre-alpha", dtype="auto")
tok = AutoTokenizer.from_pretrained("jasonjimnz/tyness-boe-1.2b-pre-alpha")
messages = [
    {"role": "system", "content": "Eres un jurista y documentalista experto en el ordenamiento jurídico español y en el BOE. Responde con rigor técnico y cita identificadores oficiales (BOE-A-AAAA-NNNNN). Si un dato no consta, decláralo explícitamente."},
    {"role": "user", "content": "¿Cuál es el identificador BOE del Decreto-ley 9/2020?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

Files

Limitations

• Hallucinates norm-specific facts and citations — verified in evaluation. Never use for real legal guidance.
• Trained on 10 norms only: outside that scope it answers in-style but without grounding.
• Base (non-instruct) LFM2.5 lineage: better at completion-style tasks than chat.
• Spanish only.

Attribution

• Base model: LiquidAI (LFM Open License v1.0).
• Data: Agencia Estatal Boletín Oficial del Estado, reused per its open-data conditions (Resolución de 27 de junio de 2024), via legalize-es.

Trained and published with Tyness Train v0.4.7 — Pre-Alpha, October 2026.