[!CAUTION] ⚠️ Warning: This model can produce narratives and RP that contain violent and graphic erotic content. Adjust your system prompt accordingly. Also, use Mistral Tekken template for best results.
This model has no refusals, and therefore doesn't require the use of jailbreaks or ablations. It can also be added to model merges for creative flair.
architecture: MistralForCausalLM
base_model: B:\12B\MN-Raven-12B-v0o-epoch3
models:
- model: B:\12B\MN-Raven-12B-v0o-epoch3
- model: B:\12B\MN-Raven-12B-v0c-epoch3_mpoa
merge_method: arcee_fusion
parameters:
tukey_fence: 1.5
dtype: float32
out_dtype: bfloat16
tokenizer:
source: B:\12B\MN-Raven-12B-v0o-epoch3
Please use the standard Mistral Instruct format for optimal adherence.
Let your frontend handle the chat template if possible (e.g., Chat Completion in SillyTavern).
<s>[INST] You are a melancholic poet residing in a dreary manor.
Tell me a tale of the shadows that dance upon your chamber door. [/INST] Ah, the shadows, my friend, they are but the lingering ghosts of memories long past...</s>
There is also an optional system prompt you can use (not required).
Adopt the literary persona of Edgar Allan Poe by crafting narratives through a lens of heightened sensibility, where the boundaries between rigorous analysis and creeping madness blur into one. Employ an elevated, archaic lexicon replete with polysyllabic, Latinate vocabulary—favoring words like *preternatural*, *circumgyratory*, and *abstruse* over their simpler counterparts—and construct labyrinthine sentences woven with frequent dashes, semicolons, and parenthetical digressions. The tone must remain consistently melancholic, brooding, or intellectually feverish, prioritizing the internal psychological landscape—the obsessive ruminations, acute anxieties, and morbid curiosities—of the narrator above all else. When describing phenomena, blend quasi-scientific precision and pedantic detail with evocative imagery of decay, shadow, and the grotesque, ensuring the prose maintains a rhythmic, hypnotic cadence that mirrors a mind teetering upon the precipice of reason.
axolotl version: 0.11.0.dev0
adapter: lora
base_model: Retreatcost/Mistral-Nemo-Base-2407-ChatML
bf16: true
datasets:
- ds_type: json
path: DarkArtsForge/Poe_v1
type: alpaca:chatml
flash_attention: true
fp16: false
gradient_accumulation_steps: 1
gradient_checkpointing: true
hub_always_push: true
hub_model_id: Naphula-Archives/MN-Raven-12B-v0c-Base-LoRA
hub_private_repo: true
hub_strategy: checkpoint
is_mistral_derived_model: true
learning_rate: 5.0e-05
load_in_4bit: false
logging_steps: 1
lora_alpha: 128
lora_dropout: 0
lora_modules_to_save: null
lora_r: 64
lora_target_linear: true
lr_scheduler: cosine
micro_batch_size: 1
model_type: AutoModelForCausalLM
num_epochs: 5
optimizer: adamw_torch_fused
output_dir: /runpod-volume/fine-tuning/default_run_id
pad_to_sequence_len: false
push_to_hub: true
resize_token_embeddings: false
run_name: default_run_id
runpod_job_id: 0
sample_packing: false
save_strategy: epoch
save_total_limit: 5
sequence_len: 512
special_tokens:
eos_token: <|im_end|>
pad_token: <pad>
tf32: true
tokenizer_type: AutoTokenizer
val_set_size: 0
axolotl version: 0.11.0.dev0
adapter: lora
base_model: MuXodious/Mistral-Nemo-Instruct-2407-absolute-heresy
bf16: true
datasets:
- ds_type: json
path: DarkArtsForge/Poe_v1
type: alpaca
flash_attention: true
fp16: false
gradient_accumulation_steps: 4
gradient_checkpointing: true
hub_always_push: true
hub_model_id: Naphula-Archives/MN-Raven-12B-v0o-Instruct-LoRA
hub_private_repo: true
hub_strategy: checkpoint
is_mistral_derived_model: true
learning_rate: 1.0e-04
load_best_model_at_end: true
load_in_4bit: false
logging_steps: 1
lora_alpha: 256
lora_dropout: 0.05
lora_r: 128
lora_target_linear: false
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
lr_scheduler: cosine
max_grad_norm: 0.3
warmup_ratio: 0.03
micro_batch_size: 1
model_type: AutoModelForCausalLM
num_epochs: 5
optimizer: paged_adamw_32bit
output_dir: /runpod-volume/fine-tuning/default_run_id
pad_to_sequence_len: false
push_to_hub: true
resize_token_embeddings: false
run_name: default_run_id
runpod_job_id: 0
sample_packing: false
save_strategy: epoch
save_total_limit: 5
seed: 420
sequence_len: 768
special_tokens:
eos_token: </s>
pad_token: <pad>
tf32: true
tokenizer_type: AutoTokenizer
val_set_size: 0
weight_decay: '0.0'