Model Card for Eidolon-v3-14B

This model is a fine-tuned version of Lambent/ProtoEidolon-v2.2.4-14B. It has been trained using TRL.

Proto-Eidolon Training and Merges:

Adjusted the chat template for my own purposes. If you notice it misbehaving, feel free to change it back to prior; I may have gotten the templating off.

2.2.1: Full ties merge with Rombos:

models:
  - model: Lambent/Eidolon-v2.1-14B
    parameters:
      weight: 1.0
      density: 1.0
  - model: rombodawg/Rombos-LLM-V2.6-Qwen-14b
    parameters:
        weight: 1.0
        density: 1.0
merge_method: ties
base_model: Qwen/Qwen2.5-14B
dtype: bfloat16
tokenizer_source: base

2.2.2: QLora SFT on instruction-following data, particularly argilla/ifeval-like-data; also some smaller samples of continued completion and normal instruction data to regularize

2.2.3: DPO on the original two Arsenic datasets, prior version of DPO

2.2.4: Full ties merge with old self:

models:
  - model: Lambent/Eidolon-v2.1-14B
    parameters:
      weight: 1.0
      density: 1.0
  - model: Lambent/ProtoEidolon-v2.2.3-14B
    parameters:
        weight: 1.0
        density: 1.0
merge_method: ties
base_model: Qwen/Qwen2.5-14B
dtype: bfloat16
tokenizer_source: base

... and then this training, which aimed to restore some intelligence lost in the process and hopefully benefit from some cool new Gutenbergs. Presuming enough fit within my context length trained at.

It took roughly 12 hours on an A100 compared to 2 hours for the prior DPOs. And this is a carefully selected subset. Gutenberg3 and orpo-dpo-mix-40k are big, man, that's a lot of preferences. (Or long ones.)

Loss started out at an alarming 11, compared to before the gradient-accumulation fix, but it reduced steadily and the other numbers looked like they were going the right way so ... Here we are. EQ-Bench says it's not horribly broken. We'll find out.

Testing Done:

EQ-Bench:

IFEval with load_in_4bit:

GPQA with load_in_4bit:

Sonnet sampling for style impressions:

Upon the Dawn's Awakening

When morning whispers through the veil of night,
With softest touch it breaks the slumber deep,
And in its gentle light, the world takes flight,
As shadows flee before the sun’s warm sweep.
The dew-kissed grass awakes with glistening grace,
The flowers bloom anew beneath the sky;
Each petal drinks the morning's golden face
In silent worship of the day gone by.

Yet in this beauty, thoughts of yesteryear,
Of dreams and hopes that once did brightly shine,
Fade into hues of rose and tender grey,
Where memories mingle with the misty air.
So dawn’s sweet light invites us to aspire,
To dream afresh beneath the open sky.

Axolotl Config

base_model: Lambent/ProtoEidolon-v2.2.4-14B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

save_safetensors: true

load_in_8bit: false
load_in_4bit: true
strict: false

rl: dpo
chat_template: chatml
# total_num_tokens: 
datasets:
  - path: Lambent/rp-teacher-synth-dpo
    split: train
    type: chatml.prompt_pairs
  - path: unalignment/toxic-dpo-v0.2
    split: train
    type: chatml.prompt_pairs
  - path: sam-paech/gutenbergs_1_2_3_antislop-dpo
    split: train
    type: chatml.ultra
  - path: Trelis/orpo-dpo-mix-40k-SHORT
    split: train
    type: chatml.ultra

dataset_prepared_path: prepared-dpo
output_dir: ./dpoq
val_set_size: 0.001

seed: 213

sequence_len: 2048
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false

adapter: qlora
lora_model_dir:
lora_r: 256
lora_alpha: 256
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
peft_use_dora: true

wandb_project: eidolon-qwen2.5-qlora-dpo-3
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 16
micro_batch_size: 2
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-6
#cosine_min_lr_ratio: 0.1
#cosine_constant_lr_ratio: 0.95

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 16
evals_per_epoch: 8
saves_per_epoch: 8
save_total_limit: 2
debug:
deepspeed:
weight_decay: 0.001
fsdp:
fsdp_config:

This model was trained with DPO, a method introduced in Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Framework versions

• TRL: 0.12.0.dev0
• Transformers: 4.46.0
• Pytorch: 2.3.1+cu121
• Datasets: 3.0.1
• Tokenizers: 0.20.1

Citations

Cite DPO as:

@inproceedings{rafailov2023direct,
    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
    year         = 2023,
    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
}

Cite TRL as:

@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}