RicardoEstep/RPBizkitRemiX-v2-12B

"Experimental Mistral Nemo Mix" using... "DARE TIES + A Custom Python Script, merges using Mergekit".

This is a "RemiX" of some of my "very best" models in one. After trying it, It's a kind of "hyperactive" model (as V2), + It has "a kind of humanity" (as V9), and sometimes does NOT act like a completly "uncensored" model, but stils just "f**k*ng scolds" you back (as V6)... 😈🥲

¡This merge is based in 2 parts!

Part One: "The Merge".

Models Merged:

• natong19/Mistral-Nemo-Instruct-2407-abliterated # Base!
• RicardoEstep/RPBizkit-v2-12B
• RicardoEstep/RPBizkit-v6-12B
• RicardoEstep/RPBizkit-v9-12B

Configuration Used:

merge_method: dare_ties
base_model: natong19/Mistral-Nemo-Instruct-2407-abliterated
tokenizer_source: base
parameters:
  normalize: true
  int8_mask: false
  random_seed: 5318008
models:
  - model: RicardoEstep/RPBizkit-v9-12B
    parameters:
      weight: 0.30
      density: 0.45
  - model: RicardoEstep/RPBizkit-v6-12B
    parameters:
      weight: 0.59
      density: 0.7
  - model: RicardoEstep/RPBizkit-v2-12B
    parameters:
      weight: 0.11
      density: 0.22
dtype: bfloat16

Part Two: "The Light Re-Abiliteration" Script".

Models Merged:

• .\output3 # Merge.
• nbeerbower/Mistral-Nemo-12B-abliterated-LORA

Script Used:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# -------- Configuration --------
base_model_path = "./output3"
tokenizer_path = "./output3"
lora_path = "nbeerbower/Mistral-Nemo-12B-abliterated-LORA"
output_path = "./output4"

# Hybrid scaling
ATTENTION_SCALE = 0.12  # It's practically a "small" cleaning for "V2 Version".
MLP_SCALE = 0.06
VOCAB_SCALE = 0.03

EXPECTED_VOCAB_SIZE = 131072

# --------------------------
print("Loading base model...")

model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
)

model.config.use_cache = False

# --- Fix Embeddings ---
print("Checking embeddings...")

current_vocab_size = model.get_input_embeddings().weight.shape[0]

if current_vocab_size != EXPECTED_VOCAB_SIZE:
    print(f"Resizing embeddings from {current_vocab_size} → {EXPECTED_VOCAB_SIZE}")
    model.resize_token_embeddings(EXPECTED_VOCAB_SIZE)

model.tie_weights()

# --- APPLY LoRA MANUALLY ---
print("Loading LoRA...")

model = PeftModel.from_pretrained(
    model,
    lora_path,
    adapter_name="default"
)

# --- Hybrid Scaling ---
print("Applying hybrid scaling...")

adapter_name = "default"

attention_layers = ["q_proj", "k_proj", "v_proj", "o_proj"]
mlp_layers = ["up_proj", "down_proj", "gate_proj"]
vocab_layers = ["lm_head"]

for name, module in model.named_modules():

    # Only modify real LoRA layers
    if hasattr(module, "lora_A") and hasattr(module, "lora_B"):

        if any(layer in name for layer in attention_layers):

            if hasattr(module, "scaling"):
                module.scaling[adapter_name] = ATTENTION_SCALE

        elif any(layer in name for layer in mlp_layers):

            if hasattr(module, "scaling"):
                module.scaling[adapter_name] = MLP_SCALE

        elif any(layer in name for layer in vocab_layers):

            if hasattr(module, "scaling"):
                module.scaling[adapter_name] = VOCAB_SCALE

# --- Merge LoRA ---
print("Merging LoRA into base weights...")

model = model.merge_and_unload(progressbar=True)

# --- Load Tokenizer ---
print("Loading tokenizer...")

tokenizer = AutoTokenizer.from_pretrained(
    tokenizer_path,
    trust_remote_code=True
)

# --- Save Final Model ---
print("Saving final model...")

model.save_pretrained(
    output_path,
    safe_serialization=True
)

tokenizer.save_pretrained(output_path)

print("Hybrid LoRA merge complete!")

Final Notes;

The Tokenizer & Embedding Sizes (131072) are clean + It's "supposed to" support now a full "128K Context Size". The recommended chat template to use is "Alpaca (with "RAW" inputs)". The configuration files are already "Tweaked", to don't use any "Chat Template". **It's recommended to use Lower Temps ("1.15, max." - "0.86 > 0.92 recommended", for "Creative RPs").

• Quantized Model, Here RicardoEstep/RPBizkitRemiX-v2-12B-GGUF.