Occitan Gemma 3 4B IT (RS-LoRA, Merged)
This repository contains a fine-tuned version of Google's Gemma 3 4B Instruct optimized for the Occitan language using RS-LoRA (Rank-Stabilized Low-Rank Adaptation).
Repository Structure
- Root Directory: Full merged Safetensors weights (bfloat16, compatible with
transformers) adapter/Folder: The raw RS-LoRA adapter files (for use with PEFT)gguf/Folder: Quantized GGUF versions (f16, Q2_K, Q4_K_M, Q5_K_M, Q8_0) for local inference via llama.cpp, LM Studio, Ollama, etc.
Usage
Using the merged model (Transformers)
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "julienp79/occitan-gemma-3-4b-it-rslora-sfttrainer"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [
{"role": "user", "content": "Escrivètz un cort paragraf en occitan sus la lenga occitana e son importància."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Using the RS-LoRA adapter with PEFT
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = AutoModelForCausalLM.from_pretrained(
"google/gemma-3-4b-it",
device_map="auto",
torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")
# Load adapter from the adapter/ subfolder
model = PeftModel.from_pretrained(base_model, "julienp79/occitan-gemma-3-4b-it-rslora-sfttrainer", subfolder="adapter")
model.eval()
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "Sètz un assistent utila que parla occitan."}],
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Using GGUF quantizations (llama.cpp)
llama-cli -hf julienp79/occitan-gemma-3-4b-it-rslora-sfttrainer:Q4_K_M \
-p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
-n 256 -e --temp 0.7
Or via the GGUF file directly:
llama-cli -m gguf/occitan-gemma-3-4b-it-rslora-sfttrainer-Q4_K_M.gguf \
-p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
-n 256 -e --temp 0.7
Adapter Details
- Adapter Type: RS-LoRA (Rank-Stabilized LoRA,
use_rslora=True) - Base Model: google/gemma-3-4b-it (Google Gemma 3 4B Instruct)
- LoRA Rank (r): 32
- LoRA Alpha: 32
- Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Dropout: 0
- Training Method: SFT (Supervised Fine-Tuning) with SFTTrainer
Training Details
- Block Size: 768 tokens
- Learning Rate: 1e-4 (cosine schedule)
- Optimizer: Paged AdamW 8-bit
- Max Gradient Norm: 1.0
- Warmup Steps: 200
- Weight Decay: 0.01
- Epochs: 5
- Total Steps: ~1270
- Final Loss: ~1.18
- Hardware: RTX 3060 12GB (full bfloat16 base model)
Training procedure
This model was trained with SFT (Supervised Fine-Tuning) using SFTTrainer on raw Occitan text (literary, journalistic, grammar, and encyclopedic sources). No chat templating was applied during training — the model learns language structure via causal language modeling on chunked text.
Framework versions
- PEFT 0.19.1
- TRL: 1.4.0
- Transformers: 5.8.1
- Pytorch: 2.6.0+cu124
- Datasets: 4.8.5
- Tokenizers: 0.22.2