AstraGPT-7B 🚀

A 7-Billion Parameter Decoder-Only Language Model

Custom BPE Tokenizer Training · Reasoning Fine-Tuned with LoRA on Dual RTX 4090

Built by Aditya Wakharkar | Tantra AI Labs | Portfolio

🧠 What is AstraGPT-7B?

AstraGPT-7B is a 7-billion parameter decoder-only language model built and trained by Tantra AI Labs — including a custom BPE tokenizer trained from scratch (64k vocabulary, later mapped to the 152k runtime vocab) and a full supervised fine-tuning pipeline written in PyTorch.

The model was reasoning fine-tuned via LoRA on a private VPS with 2× NVIDIA RTX 4090 GPUs (48GB VRAM), on a self-generated dataset of 193,841 instruction–reasoning–answer triples (astragpt-reasoning-dataset), giving it native <think>...</think> chain-of-thought support.

"Most people fine-tune models. We built the whole pipeline — tokenizer, training loop, and all."

For the coding-specialized sibling, see AstraGPTCoder-7B.

🏗️ Architecture

Decoder-only transformer, LLaMA-family design (Qwen2-compatible layout):

🔬 Training & Fine-Tuning

🤔 Reasoning Support

Native <think> chain-of-thought:

Input:  What is 15 * 47?

Output:
<think>
15 × 47 = 15 × 40 + 15 × 7 = 600 + 105 = 705
</think>
705

Trigger explicitly by appending "<think>\n" after the chat template.

⚡ Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "tantra-ai-labs/AstraGPT-7B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
)

messages = [
    {"role": "system", "content": "You are AstraGPT, a helpful AI by Tantra AI Labs. Think with <think>...</think> before answering."},
    {"role": "user", "content": "Explain quantum tunneling simply."},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) + "<think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=1024, temperature=0.3,
                         do_sample=True, repetition_penalty=1.1,
                         pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

4-bit quantized (~6GB VRAM):

from transformers import BitsAndBytesConfig
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb, device_map="auto")

⚠️ Limitations

• 7B scale — complex multi-step math and long-horizon reasoning may fail
• Can produce confident but incorrect answers — verify critical outputs
• English-primary performance
• <think> reasoning works best with an explicit <think>\n prompt prefix

🌱 Environmental Impact

• Hardware: 2× RTX 4090 (48GB), private bare-metal VPS
• Training duration: ~3–4 hours
• Estimated emissions: ~2–3 kgCO₂eq

📚 Related Resources

• 📦 Datasets: astragpt-reasoning-dataset (193,841 triples) · high-reasoning-dataset-v1 (2,139 Q&A)
• 💻 Code sibling: AstraGPTCoder-7B
• 🐙 GitHub: github.com/codewith-aditya · Tantra AI Labs
• 🌐 Portfolio: codwith-aditya.pages.dev
• 🔌 Production serving: Astra AI Gateway — 40+ models, HTTP/2 streaming

📖 Citation

@misc{astragpt7b2026,
  author       = {Aditya Wakharkar},
  title        = {AstraGPT-7B: A 7B LLM with Custom Tokenizer and Chain-of-Thought Reasoning},
  year         = {2026},
  publisher    = {HuggingFace},
  organization = {Tantra AI Labs},
  url          = {https://huggingface.co/tantra-ai-labs/AstraGPT-7B},
  note         = {Custom BPE tokenizer, GQA 28Q/4KV, RoPE, SwiGLU, LoRA reasoning fine-tune on 2× RTX 4090}
}