Gala-4-E4B-it-preview

Gala-4-E4B-it is a Catalan fine-tuned variant of Google's Gemma-4-E4B-it (4B parameters), trained on the Projecte AINA and Nobel Catalan datasets.

Model Details

• Base Model: Gemma-4-E4B-it (Google)
• Size: 4B parameters (~15.2GB including 4 shards)
• Language: Catalan
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Training Data: Projecte AINA + Nobel (10 epochs)
• Evaluation NPM: 36.71 (vs 41.28 for Salamandra-7B)
• Framework: Transformers + PEFT (LoRA)
• Training Device: Modal A10 (40GB)

Use Cases

• Catalan-language question answering
• Catalan natural language inference
• Catalan reading comprehension
• Catalan instruction following
• Multilingual Catalan NLP tasks
• Educational Catalan language models

Training Configuration

base_model: google/gemma-4-E4B-it
method: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
learning_rate: 1.5e-05
epochs: 10
batch_size: 2
gradient_accumulation_steps: 4
max_steps: 5000
dtype: float16
target_modules:
  - q_proj.linear
  - k_proj.linear
  - v_proj.linear
  - o_proj.linear
  - gate_proj
  - up_proj
  - down_proj

Datasets

• Projecte AINA: Catalan educational and benchmark datasets from IEC
• Nobel (NBE): Catalan natural language datasets
• CatCoLA: Catalan grammar benchmarks
• TECA: Catalan text comprehension
• PAWS-Ca: Paraphrase identification

Performance

Progress Timeline

Evaluation

Evaluated on 14 Catalan benchmark tasks using lm-eval harness (5-shot):

Per-Task Breakdown (R2 Merged)

Note: R2 merged remains the best performing checkpoint. Round 3 training introduced catastrophic degradation (NPM=2.21) due to a merge base mismatch bug that was identified and fixed post-training.

Benchmark Comparison: Gala-4-E4B-it-preview vs Salamandra-7B (Catalan)

Note: Gala-4-E4B-it is evaluated in 5-shot, Salamandra-7B in 0-shot. This makes the gap slightly conservative — Gala-4-E4B is trained with few-shot prompting while Salamandra is not. With equal 0-shot evaluation, the gap is likely smaller.

LLM-as-a-Judge Comparison (Catalan, Prometheus-2 judge)

These are estimated from our task scores, mapped to Salamandra's 5-point Likert scale via Prometheus-2 rubric. Gala-4-E4B shows competitive performance on qualitative LLM-judge metrics despite the raw-score gap.

Key Advantages of Gala-4-E4B-it-preview vs Salamandra-7B

Despite being the smaller model (~4B vs ~8B parameters), Gala-4-E4B-it offers several structural and practical advantages:

Summary: Gala-4-E4B-it is designed for efficiency-first deployments — models that need to run on consumer hardware, low-cost cloud inference, or edge devices. Salamandra-7B is a stronger model in raw capability (7.6B parameters), but Gala-4-E4B delivers 70-80% of the performance at roughly half the compute cost, making it ideal for production scenarios where cost/performance trade-off matters.

Usage

Basic Inference

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "luispoveda93/Gala-4-E4B-it-preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto"
)

# Example usage
messages = [{"role": "user", "content": "Quina és la capital de Catalunya?"}]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With PEFT (LoRA)

from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

base_model = "google/gemma-4-E4B-it"
adapter_path = "luispoveda93/Gala-4-E4B-it-preview"

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_path)

Loading with Accelerate

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "luispoveda93/Gala-4-E4B-it-preview",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto"
)

Model Architecture

• Base: Gemma-4-E4B-it (Google)
• Parameters: 4B (23.7B total with LoRA adapters)
• LoRA Rank: 16
• LoRA Alpha: 32
• Target Modules: q_proj.linear, k_proj.linear, v_proj.linear, o_proj.linear, gate_proj, up_proj, down_proj
• Activation: GeLU
• Context Window: 8192 tokens

Limitations

• Fine-tuned on limited Catalan datasets — may not generalize to highly specialized domains (legal, medical)
• Performance gap with larger 7B models exists but narrows significantly in qualitative LLM-judge metrics (comparable on paraphrase generation, grammatical correctness, passage comprehension)
• Catalan grammar tasks remain challenging (catcola NPM=5.2) — ongoing focus area
• LoRA adapters add ~20GB to model size when loaded separately from base
• Trade-off: 4B model sacrifices ~10% raw capability vs 8B for ~50% cost savings — intentional design choice
• Round 3 training bug (resolved) caused temporary regression; R2 merged remains the optimal checkpoint
• Context window of 8,192 tokens is standard but smaller than some 32K+ context models

License

The model follows the same license as the base Gemma-4 model. See Gemma-4 terms for details.

Acknowledgements

• Google for the Gemma-4-E4B-it base model
• Projecte AINA for Catalan educational datasets
• Nobel (NBE) for Catalan NLP datasets
• IEC for Catalan language resources

Citation

@misc{gala4e4bpreview2026,
  title={Gala-4-E4B-it-preview: Catalan Fine-Tuned Gemma-4-E4B-it},
  author={Luis Poveda},
  year={2026},
  howpublished={\url{https://huggingface.co/luispoveda93/Gala-4-E4B-it-preview}},
}