🚀 Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax - GGUF

This repository contains GGUF format quantizations of the distilled model, built upon Qwen/Qwen3-4B-Thinking-2507. The distilled model delivers stronger coding, reasoning, and constraint-following capabilities in a compact 4B-parameter footprint.

75.7% of benchmark questions improved, with gains up to +93% on difficult algorithmic tasks and +33.2% on constraint-heavy coding.

The model excels at multi-step reasoning, code generation, mathematical problem-solving, structured logical inference, and strict instruction-following — making it ideal for developers who want stronger technical behavior from a compact model.

📦 Model Files

Recommendation: Start with Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax-q8_0.gguf for the best performance while still remaining practical on most hardware.

🚀 Usage with llama.cpp

Option 1: Direct from Hugging Face

llama-cli \
  --model hf:<your-username>/<your-repo-name>/Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax-q8_0.gguf \
  --prompt "Write a Python function to compute the Fibonacci sequence" \
  --threads 8 \
  -no-cnv

Option 2: Local file

./llama-cli \
  --model Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax-q8_0.gguf \
  --prompt "Explain the concept of recursion in simple terms" \
  --threads 8

Option 3: With GPU offloading

./llama-cli \
  --model Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax-q8_0.gguf \
  --n-gpu-layers 40 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --threads 16 \
  --prompt "Solve: If x + y = 10 and 2x - y = 5, find x and y" \
  -no-cnv

Option 4: Chat mode with system prompt

./llama-cli \
  --model Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax-q8_0.gguf \
  --system "You are a helpful coding assistant." \
  --prompt "How do I reverse a linked list in Python?" \
  -no-cnv

🐍 Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax-q8_0.gguf",
    n_ctx=8192,
    n_threads=8,
    n_gpu_layers=40,  # Set to 0 for CPU-only
    verbose=False
)

output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Write a quicksort implementation in Python."}
    ],
    max_tokens=1024,
    temperature=0.7
)

print(output["choices"][0]["message"]["content"])

🔧 Conversion Details

These GGUF files were generated using llama.cpp via the GGUF-my-repo space.
The files are compatible with:

• llama.cpp (all recent versions)
• LM Studio
• Jan
• koboldcpp
• Any GGUF-compatible runtime

📊 Benchmark Details

• 148 questions across 8 domains
• Same prompts evaluated on both base and distilled models
• Base model: Qwen/Qwen3-4B-Thinking-2507
• Metrics: Relevancy + Similarity

Biggest Wins:

The strongest improvements are concentrated in technical reasoning, coding, mathematics, and constraint-heavy tasks.

💡 When to Use This Model

This distilled model is particularly effective for:

• 🧠 Multi-step reasoning tasks
• 💻 Code generation and debugging
• 📐 Mathematical problem-solving
• 🎯 Strict instruction and constraint following
• 🔍 Structured logical inference

📝 Notes

• The model uses the Qwen chat template (<|im_start|>system, <|im_start|>user, <|im_start|>assistant)
• Recommended context length: 8192 tokens
• The -no-cnv flag disables auto-conversation mode for single-turn prompts

📄 License

This model is released under the Apache-2.0 license.

🙏 Acknowledgments

• Base model: Qwen/Qwen3-4B-Thinking-2507
• Quantization tools: llama.cpp
• GGUF conversion: GGUF-my-repo

📎 Citation

If you use this model in your research or production, please cite:

@misc{qwen3-4b-thinking-2507-qwen3.8-max-distillation-detrax-gguf,
  author = {Your Name},
  title = {Qwen3-4B-Thinking-2507-Qwen3.8-Max-Distillation-Detrax - GGUF Quantizations},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/<your-username>/<your-repo-name>}
}

Bottom Line: If your workload is coding, reasoning, mathematics, or structured technical tasks, this distilled model is worth trying.