Warlock-1.7B

Warlock-1.7B (also known as Warlock-1.7B-Fused) is a standalone, independent 1.71B parameter instruction-tuned causal language model created by fusing the internal representational capacities of two distinct model families:

• Primary Architecture / Recipient Model: SmolLM2-1.7B-Instruct
• Feature Donor Model: Llama-3.2-1B-Instruct

This model is a single, self-contained model file (model.safetensors, 3.42 GB) that runs standard Hugging Face causal language modeling out of the box with zero runtime dependencies on donor models or custom Python inference wrappers.

What Has Been Done

Standard model merging techniques typically interpolate weights directly between checkpoints of the exact same architecture, dimensions, and layer structures (e.g., fine-tuned LoRA merges, SLERP, DARE, TIES). When combining completely disparate model families with different depth topologies, vocabulary spaces, and attention configurations (such as SmolLM2's Multi-Head Attention vs. Llama-3.2's Grouped-Query Attention), direct naive averaging fails catastrophically.

In this model:

1. Multi-Depth Representation Alignment: Key representational layers between Llama-3.2-1B and SmolLM2-1.7B were aligned across their shared hidden space ($d=2048$) to create coherent latent coordinate correspondence.
2. Energy-Calibrated Weight Fusion: The structural and reasoning features of the donor model were baked directly into the recipient's projection weights (MLP down/gate/up projections and attention projections) under strict Frobenius norm preservation. This prevents latent space distortion while permanently embedding donor capabilities.
3. Standalone Architecture: Unlike dynamic interception or dual-stream inference pipelines, the resulting model is completely standalone. It executes with native C++ KV-caching, supports Flash Attention / SDPA, and runs at full GPU throughput (~30+ tokens/sec on an RTX 4050 Laptop GPU).

Complete Empirical Evaluation Suite (All Benchmarks & Scores)

Below is the complete record of every single benchmark suite and test battery evaluated across the parent models and the fused model.

Test Suite 1: Massive 100-Benchmark Evaluation Suite (1,000 Questions)

An exhaustive evaluation across 100 distinct, self-contained benchmarks (totaling 1,000 evaluation questions) under strict zero-shot / greedy decoding settings ($\text{temperature} = 0.0$):

Macro Category Overview

100-Benchmark Win/Loss Distribution:

• Benchmarks Won or Co-Won by Warlock-1.7B: 50 / 100 (50.0%)
• Benchmarks Won Exclusively by SmolLM2-1.7B: 29 / 100 (29.0%)
• Benchmarks Won Exclusively by Llama-3.2-1B: 14 / 100 (14.0%)
• Complete Three-Way Ties: 7 / 100 (7.0%)

Detailed Breakdown of All 100 Individual Benchmarks

1. Academic Knowledge: 57 MMLU Benchmarks (570 Questions)

2. Scientific Deduction: 8 ARC-Challenge Benchmarks (80 Questions)

3. Situational Commonsense: 15 HellaSwag Benchmarks (150 Questions)

4. Quantitative Logic: 10 GSM8K Benchmarks (100 Questions)

5. World Knowledge: 10 Semantic Benchmarks (100 Questions)

Test Suite 2: Strict Zero-Shot Blind HellaSwag Benchmark (20 Unseen Items)

Evaluated under frozen Pareto-optimal $\alpha^* = 0.020$ on 20 unseen items from Rowan/hellaswag:

Test Suite 3: Multi-Anchor Hub Benchmark Sweep (15 Questions)

Test Suite 4: Official Hugging Face Leaderboard Ceiling References

Quickstart & Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/SmolLlama-1.7B-Fused"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "Explain the fundamental theorem of calculus in simple terms."
messages = [
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.6, top_p=0.9)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Model Details

• Base Architecture: LlamaForCausalLM (SmolLM2-1.7B layout)
• Parameters: 1.71 Billion
• Precision: bfloat16
• Context Length: 8,192 tokens
• VRAM Requirements: ~3.4 GB VRAM for inference (runs comfortably on 4GB–6GB consumer GPUs)