QaptaanLM-0.75B: Efficient Hybrid-Attention Foundation Base Model

QaptaanLM-0.75B is a compact, high-efficiency hybrid-attention foundation language model optimized for source code synthesis, technical reasoning, and long-context code comprehension.

Engineered by stripping the visual transformer from Qwen/Qwen3.5-0.8B-Base down to 752M dense parameters, QaptaanLM achieves state-of-the-art computational and memory efficiency on consumer GPUs and edge accelerators. It couples linear-complexity recurrence layers with dense multi-head attention and was trained on KapCode-1B (1-billion-token curated code, doc, and STEM corpus with 50% Fill-in-the-Middle infilling on Google TPU v5e-8).

🌐 Model Ecosystem & Formats

Model Specification

Recommended Generation Parameters (CPT Base)

generation_config = {
    "do_sample": False,              # Greedy decoding for exact deterministic code completion
    "temperature": 0.15,             # Low temperature when sampling
    "top_p": 0.90,
    "top_k": 40,
    "repetition_penalty": 1.10,      # Prevents repetition loops
    "eos_token_id": [248044, 248046],# <|endoftext|> and <|im_end|>
    "pad_token_id": 248044
}

Quickstart & Usage

1. Standard Code Prefix Completion

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kaptaan45/QaptaanLM-0.75B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = 'def binary_search(arr: list[int], target: int) -> int:\n    """Return index of target in sorted arr, or -1 if not found."""\n    '
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.10,
        eos_token_id=[248044, 248046],
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Fill-in-the-Middle (FIM) Code Infilling

prefix = "def calculate_circle_area(radius: float) -> float:\n    \"\"\"Compute area of circle.\"\"\"\n    if radius < 0:\n        raise ValueError('Radius cannot be negative')\n"
suffix = "\n    return area\n"

# Format: <|fim_prefix|> Prefix <|fim_suffix|> Suffix <|fim_middle|>
fim_prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=48,
        do_sample=False,
        eos_token_id=tokenizer.convert_tokens_to_ids("<|fim_middle|>"),
    )

infilled = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print("Infilled Code:\n", infilled)

Dataset Attribution

Trained on KapCode-1B (1,000,013,824 tokens) across 5 curated domains:

• Source Code (35%): Multi-language code from The Stack v3.
• Technical Documentation (20%): READMEs, Markdown guides, API references.
• Function-Level Code (20%): Annotated functions from The Vault.
• High-Quality Web (15%): STEM educational web documents from FineWeb-HQ.
• Mathematical Reasoning (10%): LaTeX equations and proofs from OpenWebMath.

License

Released under the Apache 2.0 License. Upstream base model weights and architecture adapted from Qwen/Qwen3.5-0.8B-Base by the Qwen Team (Alibaba Cloud).