💨 Vapor-2B-V1.2
Vapor-2B-V1.2 is a specialized, lightweight causal language model built with a high-efficiency Hybrid Short-Conv & Full-Attention architecture. Engineered specifically for high-throughput, low-latency, and memory-constrained inference, Vapor-2B-V1.2 delivers fast token generation with an exceptionally compact memory footprint.
⚡ Key Architectural Specifications
- Model Name: Vapor-2B-V1.2
- Model Size: ~2.0 Billion parameters (1.95B linear parameters)
- Architecture: Hybrid Linear Short-Conv and Full-Attention
- Total Layers: 30 layers (22 Causal 1D Depthwise Conv layers + 8 Multi-Head Attention layers)
- Hidden Dimension: 2,048 ($D_{\text{ffn}} = 10,752$ with SwiGLU activations)
- Attention Mechanism: Grouped-Query Attention (32 Q-heads, 8 KV-heads) with Rotary Position Embeddings (RoPE)
- Context Length: Native support up to 131,072 tokens (128k context length)
- Memory Footprint: ~4.3 GB in BF16/FP16 (Runs effortlessly on consumer GPUs like RTX 3050, RTX 4060, and edge devices)
🚀 Quickstart & Inference
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Neeze/Vapor-2B-V1.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True
)
prompt = "Explain the architectural benefits of combining 1D convolutions with self-attention in language models:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.7,
top_p=0.9
)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
📋 Intended Use
Vapor-2B-V1.2 is optimized for developers needing fast, low-latency text generation, agentic reasoning, and interactive conversation on resource-limited GPU environments without sacrificing context window length.