🧟 Frankenstein-3.0: The Unified AI Entity

A production-grade multi-modal AI orchestration system with a trained routing brain, featuring 4 specialist models, 5 integrated tools, and triple-layer fault tolerance.

Modality: Text → Text • Image • Video • Audio (orchestrated across specialists) Repo weights: RouterBrain (text router, QLoRA-fine-tuned Qwen2.5-1.5B)

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     USER REQUEST                            │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  LAYER 1: Pre-Flight Regex (100% accurate, hard patterns)   │
│  • "uploaded file" → RAG    • "then" → multi-step chain     │
│  • "explain how to" → chat                                  │
└─────────────────────────────────────────────────────────────┘
                            ↓ (pass-through)
┌─────────────────────────────────────────────────────────────┐
│  LAYER 2: RouterBrain (0.3s instant routing)                │
│  • Qwen2.5-1.5B fine-tuned with QLoRA, merged to fp16       │
│  • Outputs JSON routing decisions                           │
└─────────────────────────────────────────────────────────────┘
                            ↓ (if brain fails)
┌─────────────────────────────────────────────────────────────┐
│  LAYER 3: Qwen3.5-4B Resident (smart fallback)              │
│  • Understands complex/novel requests                       │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  SPECIALISTS:  🧠 Chat   💻 Code   🎨 Image   🎬 Video       │
└─────────────────────────────────────────────────────────────┘

The RouterBrain (This Repo's Weights)

The root weights are the RouterBrain: a distilled routing model that reads user requests and outputs JSON routing decisions in ~0.3 seconds.

Load the RouterBrain

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Questionmarkboy/frankenstein-3.0")
model = AutoModelForCausalLM.from_pretrained(
    "Questionmarkboy/frankenstein-3.0",
    torch_dtype="float16",
    device_map="auto"
)

prompt = "Route to the correct specialist (chat/code/image/video/search/calc/execute/rag). USER: Draw a robot then animate it dancing. Respond ONLY with JSON."

text = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=120, do_sample=False, repetition_penalty=1.05)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Expected Output

{"specialist": "multi", "confidence": 0.95, "reasoning": "Chained multi-step task", "steps": [{"specialist": "image", "prompt": "Draw a robot"}, {"specialist": "video", "prompt": "animate it dancing"}]}

The Full System (app.py)

Specialists

Tools

• 🔍 Web Search — DuckDuckGo real-time search
• 🧮 Calculator — SymPy safe mathematical evaluation
• ⚡ Code Execution — Sandboxed Python execution (5s timeout)
• 📚 RAG — Document upload + keyword search
• 🧠 Memory — SQLite persistent conversation history

Installation

Prerequisites

Step 1 — Clone the repository

git lfs install
git clone https://huggingface.co/Questionmarkboy/frankenstein-3.0
cd frankenstein-3.0

Step 2 — Create a virtual environment (recommended)

python -m venv venv
source venv/bin/activate        # Linux / macOS
venv\Scripts\activate          # Windows

Step 3 — Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

Step 4 — Launch the app

python app.py

Then open http://localhost:7860 in your browser.

CPU-only machines

Installation is identical. The app auto-detects the absence of CUDA and runs the RouterBrain, chat and code specialists on CPU. Image and video specialists respond with a friendly "GPU required" message instead of crashing.

Troubleshooting

• bitsandbytes warnings on CPU — expected and safely ignored by the app.
• Slow clone — the RouterBrain weights are ~3 GB; make sure Git-LFS is installed (Step 1).
• Port 7860 already in use — change server_port at the bottom of app.py.
• Out of memory on GPU — the app loads specialists on demand and unloads them after use; close other GPU processes or reduce image resolution in app.py.

Hardware Requirements

The app auto-detects CUDA. On CPU-only machines, chat/code work and image/video return a friendly "GPU required" message.

Training Details

RouterBrain Training

• Base Model: Qwen2.5-1.5B-Instruct
• Fine-tuning: QLoRA (r=32, α=64, dropout=0.05)
• Dataset: ~10k synthetic routing examples (combinatorial diversity, noise injection, 30+ edge-case patterns)
• Training: 2-3 epochs on T4 GPU
• Merge: Clean fp16 merge (no quantization rounding)

Performance Metrics

• Pre-flight layer: 100% accuracy on hard patterns
• RouterBrain: 60% standalone accuracy on diverse test set
• Effective routing: 90%+ via triple-layer fault tolerance

Multi-Step Task Chaining

User: "Draw a robot then animate it dancing"
  ↓
Pre-flight: detects "then" → routes to multi specialist
  ↓
Execution: image specialist → video specialist (sequential)
  ↓
Output: generated image + animated video with audio

Architecture Decisions

Why Triple-Layer Routing?

1. Speed: Pre-flight regex is instant (0.001s)
2. Efficiency: RouterBrain is fast (0.3s) vs prompt-based (4-6s)
3. Robustness: 4B fallback catches edge cases
4. Production-ready: Graceful degradation under all conditions

Why Distill A Routing Brain?

• 20x faster than prompt-based routing
• 50% less VRAM than using the 4B resident for routing
• Perfect JSON output (trained to only output JSON)
• Cheaper to deploy at scale

Companion Models

• Frankenstein-2.0 — Merged Mistral-7B checkpoint

Technical Stack

• Framework: HuggingFace Transformers + Diffusers
• Quantization: BitsAndBytes (NF4, 4-bit)
• UI: Gradio 4.44
• Memory: SQLite
• Training: PEFT + LoRA

Known Limitations

• Image/video generation requires 16GB+ VRAM
• RAG only supports .txt files (no PDF/DOCX yet)
• Web search is basic (DuckDuckGo HTML scraping)
• Code execution has 5s timeout for safety

Citation

@misc{frankenstein3,
  author = {Questionmarkboy},
  title = {Frankenstein-3.0: Multi-Modal AI Orchestration System},
  year = {2024},
  publisher = {HuggingFace},
  journal = {HuggingFace Repository},
  howpublished = {\url{https://huggingface.co/Questionmarkboy/frankenstein-3.0}}
}

License

Apache 2.0

Acknowledgments

• Base models: Qwen, Stability AI, MiniMax
• Frameworks: HuggingFace, Gradio, BitsAndBytes
• Training infrastructure: Kaggle T4 GPUs

Built with ❤️ for the AI community