Gemma 4 E4B · Basque GEC (Explainable)

Fine-tuned Google Gemma 4 E4B for Basque (euskara) grammatical error correction with structured, explainable JSON output. Each correction comes with a category, a normative reference, and a Basque-language explanation — enabling runtime filtering by error type, severity, or normative vs. editorial status.

Model details

Training data

Trained on horkonpon-corpus — an EBE-grounded, license-hygienic Basque GEC training corpus:

• 208,051 records (83,121 error + 88,009 clean + 36,921 suggestion)
• 199,662 training / 1,088 eval (gold split, zero document overlap)
• Grounded in EBE (Euskara Batuaren Eskuliburua, Euskaltzaindia's normative handbook) — the sole normative authority for Basque
• 16 deterministic error injectors across 9 categories
• CC-BY-SA 4.0 (commercially usable)
• Clean text from Latxa-Corpus-v2

Note: The horkonpon-corpus dataset card is not yet published on the HuggingFace Hub. It is available on GitHub at itzune/horkonpon-corpus and will be published as itzune/horkonpon-corpus (Track B) in a future release.

The model was trained on Format 2 (explain-then-correct): given a sentence with errors, the model outputs a JSON object with the corrected sentence and a list of edits, each annotated with structured metadata.

Output format

{
  "target": "Argazkiak bidaltzen dituzuen guztion artean Errigorako bi otar zozkatuko ditugu.",
  "edits": [
    {
      "src_content": "bidaltsen",
      "tgt_content": "bidaltzen",
      "category": "spelling",
      "error_type": "char_swap",
      "error_severity": 2,
      "nature": "error",
      "ref": "EBE id=1317",
      "error_description": "«ts» eta «tz» nahasketa. Hemen «tz» da zuzena. Ordezkatu bidaltsen → bidaltzen."
    }
  ]
}

Fields

Categories

capitalization, word_level, spelling, zalantza (word-choice), terminology, morphology, punctuation, proper_noun, calque

Runtime filtering

The structured output lets the application filter at inference time:

# Only apply EBE normative corrections, ignore Berria editorial suggestions
edits = [e for e in output["edits"] if e["nature"] == "error"]

# Or ignore a specific underperforming category
edits = [e for e in output["edits"] if e["category"] != "calque"]

Evaluation

Evaluated on the horkonpon-corpus gold eval split: 1,088 errorful records (1,037 error + 51 suggestion) and 1,088 clean records (false-positive test). Greedy decoding (do_sample=False), max_new_tokens=256, batch_size=32. Compared against gector-eus-v2 (GECToR, RoBERTa-eus-base, 124M params) trained on the same corpus.

Headline results

Per-category recall

The full evaluation report (eval_report.json) is included in this repository.

Key findings

1. Gemma 4 beats GECToR v2 on F0.5 (+2.0) by trading precision for recall.
2. Dramatic gains on semantic categories — spelling, zalantza (word-choice), punctuation, proper_noun — where the LLM leverages world knowledge that GECToR's fixed edit vocabulary cannot.
3. GECToR v2 wins on morphology (sequence tagging excels at fine-grained inflection) and is far more conservative on clean text (1.4% FP vs 8.6% — important for a writing tool).
4. Both models remain weak on calque (lexical calques need more training pairs).
5. 100% JSON parse rate — structured output is fully reliable; every prediction was valid JSON.

Ablation: base vs fine-tuned

To isolate the contribution of fine-tuning, the base (non-fine-tuned) Gemma 4 E4B-IT model was evaluated with the identical pipeline — same system prompt, same eval data, same metrics, same greedy decoding (4-bit quantization, max_new_tokens=512). The system prompt was embedded in each eval record so the base model received identical instructions to the fine-tuned model.

The base model is unusable for GEC: F0.5=2.2 with a 97.2% false-positive rate on clean sentences. It rewrites nearly everything (817/1088 errorful sentences changed wrongly, 97.2% of clean sentences altered), generating 10,701 false-positive edits versus the fine-tuned model's 120.

Although per-category recall appears "high" on the base model (e.g. morphology 80.8%, punctuation 83.3%), this is an artifact of aggressive over-editing — the model changes so much text that it happens to hit gold edits, but at 1.8% precision the corrections are drowned in noise.

What fine-tuning achieves:

1. Minimal-edit discipline — the base LLM has the linguistic knowledge but no notion of correctional restraint; fine-tuning teaches it to change only what is wrong (+84.5 precision, −88.6 clean FP).
2. Structured output reliability — JSON parse rate rises from 92.8% to 100%; the base model occasionally breaks JSON format on long generations.
3. Error detection calibration — recall increases (+47.9) despite becoming more conservative, meaning the model learns to identify genuine errors rather than rewriting indiscriminately.

The full ablation report is saved as eval_report_base.json in this repository.

Training details

The LoRA adapter was merged into the base model in 16-bit precision for deployment (no inference-time adapter overhead). The original 4-bit LoRA adapter is preserved in the source repo for reproducibility.

Usage

With transformers

import json
import torch
from transformers import AutoProcessor, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "itzune/gemma-4-e4b-horkonpon",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("itzune/gemma-4-e4b-horkonpon")
tokenizer = processor.tokenizer

system_prompt = (
    "Zuzendu emandako euskara testuko akatsak, ahal bezain aldaketa gutxien eginez, "
    "eta azaldu edizio bakoitza. Jarraitu honako arau hau: «Euskaltzaindiaren aditz-taulak "
    "eta eranskina: aditz komunztadura, deklinabide-atxiki eta pertsona-izenordainak "
    "Euskaltzaindiak ezarritako paradien arabera erabili». Eman emaitza JSON gisa: "
    '{"target": "<testu zuzendua>", "edits": [{"src_content": "<jatorrizkoa>", '
    '"tgt_content": "<zuzena>", "category": "...", "error_type": "...", '
    '"error_severity": 1, "nature": "error", "ref": "EBE ...", '
    '"error_description": "..."}]}.'
)

user_input = "Argazkiak bidaltsen dituzuen guztion artean Errigorako bi otar zozkatuko ditugu."

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_input},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text=prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        temperature=None,
        top_p=None,
        top_k=None,
    )

response = tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
result = json.loads(response)
print(result["target"])
# Argazkiak bidaltzen dituzuen guztion artean Errigorako bi otar zozkatuko ditugu.
for edit in result["edits"]:
    print(f"  {edit['src_content']} → {edit['tgt_content']}  [{edit['category']}]")
# bidaltsen → bidaltzen  [spelling]

Note: Gemma 4 is a multimodal model. Use tokenizer.apply_chat_template(..., tokenize=False) followed by tokenizer(text=prompt, ...) to stay on the text-only path. Calling apply_chat_template(..., tokenize=True, return_tensors="pt") routes through the multimodal processor and expects a list-of-dicts content format.

Inference speed

For production, consider serving with vLLM (merged LoRA, paged attention, continuous batching) for 5–10× speedup over naive model.generate().

Limitations

• Clean FP rate (8.6%) — the model sometimes "corrects" already-correct sentences. For a writing assistant where false positives are costly, GECToR v2 (1.4% FP) may be preferable, or apply a confidence threshold via token logprobs.
• Calque recall (17.1%) — lexical calques remain the weakest category; more training pairs needed.
• Morphology (77.1%) — lower than GECToR v2's sequence-tagging approach (94.5%). The LLM occasionally over-generalizes inflection rules.
• Basque-specific — fine-tuned for Basque only; performance on other languages reflects the base model, not the fine-tune.
• Multimodal components untrained — the vision / audio / video components are frozen and untested; use text-only.

Ethical considerations

• The model corrects grammar but should not be used to gatekeep language use or discourage Basque learners.
• EBE is the sole normative authority; nature=suggestion items are editorial (Berria style guide), not normative.
• Training data is synthetic-error-injection based; the model may reflect biases in the source corpus (Latxa-Corpus-v2).

Citation

@misc{ezpeleta2026gemma4e4bhorkonpon,
  title        = {Gemma 4 E4B Fine-tuned for Basque Grammatical Error Correction},
  author       = {Ezpeleta, Xabi},
  year         = {2026},
  howpublished = {Hugging Face model},
  url          = {https://huggingface.co/itzune/gemma-4-e4b-horkonpon},
}

Related

• horkonpon-corpus — the training corpus (CC-BY-SA 4.0; HF publication pending)
• gector-eus-v2 — the GECToR model trained on the same corpus (F0.5=78.8, ONNX int4, in-browser)
• txukun — the Basque writing assistant this model powers
• google/gemma-4-E4B-it — the base model
• Latxa-Corpus-v2 — the clean-text source underlying horkonpon-corpus training data
• Euskaltzaindia — EBE — Euskara Batuaren Eskuliburua, the sole normative authority for Basque (grounds all nature=error corrections)
• Berria Estilo Liburua — Berria's editorial style guide (grounds nature=suggestion corrections)