LFM2.5-2.6B-Pashto-Zi-b1 — 342M Token LoRA Fine-Tune

A 16-bit (FP16) fine-tuned version of nassimjp/LFM2.5-2.6B-Pashto-Zi-b.

This model was fine-tuned with LoRA using Unsloth + Hugging Face TRL, with a maximum sequence length of 16,384 tokens and a total of approximately 342.4 million training tokens.

The goal of this training run was to further adapt the LFM2.5 2.6B model to the training corpus while preserving strong native-language behavior and improving its ability to process and generate complex text.

Model Details

• Developed by: nassimjp
• Base model: nassimjp/LFM2.5-2.6B-Pashto-Zi-b1
• Model family: LFM2.5
• Model size: ~2.6B parameters
• Training method: LoRA / Supervised Fine-Tuning
• Precision: 16-bit (FP16)
• Training framework: Unsloth
• Trainer: Hugging Face TRL
• License: Apache-2.0
• Primary language: Pashto
• Additional language: English
• Maximum sequence length: 16,384 tokens

Training Summary

This was a large-scale single-epoch LoRA fine-tuning run.

LoRA Configuration

The model uses a standard LoRA adapter with a rank of 32.

LoRA rank (r):       32
LoRA alpha:          64
LoRA dropout:        0
RS-LoRA:             disabled
DoRA:                disabled
LoftQ:               disabled

Target modules

LoRA adapters were applied to the following Transformer modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

This covers both the main attention projections and the MLP projections.

The configuration was intentionally kept as a conventional LoRA setup rather than using DoRA, LoftQ, or rank-stabilized LoRA.

Training Configuration

training:
  max_seq_length: 16384
  num_epochs: 1
  learning_rate: 0.00003
  embedding_learning_rate: null
  batch_size: 1
  gradient_accumulation_steps: 2
  warmup_steps: 10
  max_steps: 0
  save_steps: 3000
  eval_steps: 0.04
  weight_decay: 0.001
  random_seed: 3407
  packing: false
  train_on_completions: false
  gradient_checkpointing: unsloth
  optim: adamw_8bit
  lr_scheduler_type: linear

lora:
  lora_r: 32
  lora_alpha: 64
  lora_dropout: 0
  target_modules:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj
  use_rslora: false
  use_loftq: false
  use_dora: false

logging:
  enable_wandb: false
  enable_tensorboard: false
  log_frequency: 10

Training Results

The completed training run reached:

Steps:          75,042 / 75,042
Completion:     100%
Tokens:         342,432,352
Epoch:          1.00
Final LR:       ~4.00e-10
Final Grad Norm: 2.341

The evaluation-loss curve showed a continuous overall decrease throughout training, with no obvious late-stage reversal in the final evaluation curve.

Approximate evaluation-loss progression:

~1.87
 ↓
~1.70
 ↓
~1.58
 ↓
~1.50
 ↓
~1.43
 ↓
~1.38
 ↓
~1.34
 ↓
~1.32–1.33

The final training dashboard reported an instantaneous training loss of approximately:

1.6652

The training-loss graph itself remained noisy, as expected for token-level training, while its smoothed trend remained substantially lower than the initial loss.

Token Scale

This training run processed:

342,432,352 tokens

during one complete epoch.

The original estimate was approximately 341M tokens, and the completed run reached approximately 342.4M tokens.

This makes the run a large-scale LoRA fine-tuning experiment for a 2.6B-parameter model.

Importantly, the token count alone should not be interpreted as a universal LoRA capacity limit. LoRA performance depends on the model architecture, adapter rank, target modules, dataset quality and diversity, learning schedule, number of epochs, and evaluation behavior.

Why LoRA Rank 32?

This model uses:

r = 32
alpha = 64

rather than automatically increasing the rank based only on dataset size.

The training run successfully completed approximately 342.4M tokens while maintaining a steadily improving evaluation-loss curve.

Therefore, this model provides a practical reference point for evaluating whether a higher LoRA rank is actually beneficial for this particular base model and dataset.

Future experiments may compare configurations such as:

r=32, alpha=64
r=64, alpha=128

while keeping the dataset and other training parameters controlled.

Sequence Length

The maximum training sequence length was:

16,384 tokens

Packing was disabled:

packing: false

This means individual training examples were not packed together into a single sequence.

The 16K value is the maximum sequence length, not the average sequence length of the dataset.

Training Hardware

Training was performed locally using an NVIDIA GPU.

Observed training behavior included:

• GPU utilization: approximately 100%
• GPU memory usage: approximately 7.7 / 16 GB
• GPU temperature: approximately 69°C
• GPU power: approximately 267W / 295W
• Training duration: approximately 23 hours 21 minutes

The training run completed successfully without an observed GPU memory failure.

Frameworks

This model was trained using:

• Unsloth
• Hugging Face Transformers
• Hugging Face TRL
• LoRA / PEFT-compatible fine-tuning

Unsloth provides optimized fine-tuning workflows integrated with the Hugging Face ecosystem and supports LoRA/QLoRA and related training workflows. The resulting models remain compatible with the Hugging Face ecosystem.

Intended Use

This model is intended for experimentation and research involving:

• Pashto language generation
• Pashto instruction following
• Pashto conversational systems
• Pashto language modeling
• multilingual language adaptation
• low-resource language research
• LLM fine-tuning research
• LoRA research
• evaluation of small language models

The model may be useful as a component in Pashto-focused AI applications, research systems, and local/offline inference experiments.

Limitations

This is a fine-tuned language model and may produce:

• incorrect information
• hallucinated facts
• grammatical errors
• inconsistent reasoning
• culturally inappropriate responses
• unwanted repetition
• incorrect answers to complex questions

A lower evaluation loss does not automatically mean that the model is factually correct or universally better.

For serious applications, independent evaluation and human review are recommended.

Base Model

The model was fine-tuned from:

nassimjp/LFM2.5-2.6B-Pashto-Zi-b

Please refer to the base model card for information about the original model, architecture, tokenizer, and licensing.

License

This model is released under the:

Apache License 2.0

Please also review the licensing and usage conditions of the base model and any datasets used during fine-tuning.

Citation

If you use this model in research or another project, please cite the model repository:

nassimjp/LFM2.5-2.6B-Pashto-Zi-b1

and this fine-tuned model:

nassimjp/LFM2.5-2.6B-Pashto-Zi-b1-342M-LoRA

Acknowledgements

Special thanks to the open-source communities behind:

• Hugging Face
• Transformers
• TRL
• Unsloth
• PEFT
• LFM2 / LFM2.5

This work builds on the open-source LLM ecosystem to explore stronger and more capable Pashto language models.

Training Configuration — Compact Reference

Base Model:
nassimjp/LFM2.5-2.6B-Pashto-Zi-b

Method:
LoRA SFT

Precision:
FP16 / 16-bit

Tokens:
342,432,352

Steps:
75,042

Epochs:
1

Max Sequence Length:
16,384

Batch Size:
1

Gradient Accumulation:
2

Learning Rate:
3e-5

Scheduler:
Linear

Optimizer:
AdamW 8-bit

Weight Decay:
0.001

Warmup:
10 steps

LoRA Rank:
32

LoRA Alpha:
64

LoRA Dropout:
0

Target Modules:
q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

RS-LoRA:
False

DoRA:
False

LoftQ:
False

Packing:
False

Train on Completions:
False

Gradient Checkpointing:
Unsloth

Random Seed:
3407

Training Time:
~23h 21m

Final Reported Training Loss:
1.6652

Final Evaluation Loss:
~1.32–1.33

Final Learning Rate:
~4e-10

🦥 Made with Unsloth

This model was trained with Unsloth and Hugging Face TRL.