🐻🔥 BareTorch-500M-Base
BareTorch-500M Base is a foundational sub-quadratic language model built under a pure GEMM-compliant, kernel-free paradigm. The model combines CS-LRAD (Chunk-Segmented Low-Rank Associative Delta Engine) recurrent layers with standard Transformer multi-head self-attention in a 3:1 interleaved hybrid topology ($3\times\text{CS-LRAD} \to 1\times\text{Transformer}$).
By structuring sub-quadratic state updates into block-parallel chunk segments ($C=32$), BareTorch bypasses the compilation and hardware lock-in of custom CUDA or Triton kernels, running with $O(N)$ execution and memory efficiency natively across NVIDIA CUDA, Apple Silicon MLX, WebGPU, and TPUs.
📐 Model Architecture Specifications
- Parameters: ~500M (498.2M active parameters)
- Hidden Dimension ($d_{model}$): 1,152
- Total Layers: 24 (Interleaved 18x CS-LRAD + 6x Transformer)
- Attention Heads: 16 ($d_{head} = 72$)
- CS-LRAD Subspace Rank ($r$): 8
- Chunk Size ($C$): 32 tokens
- Tokenizer:
HuggingFaceTB/SmolLM2-360M (Vocab size: 49,152)
- Context Window: Up to 32,768 tokens
🏋️ Pre-Training Runway Specs & Loss Convergence
- Training Runway: 100 Billion Tokens ($190,735$ optimization steps)
- Hardware Cluster: $4\times$ NVIDIA H100 SXM (80GB VRAM)
- Global Batch Size: $524,288$ tokens/step ($256$ sequences of length $2048$)
- Optimizer & LR: AdamW ($ ext{LR}_{peak} = 6 \times 10^{-4}$, weight decay $0.1$, cosine decay scheduler with $2,000$ warmup steps)
Pre-Training Evaluation Loss Milestones
| Optimization Step | Tokens Processed | Train Loss | Eval Loss |
|---|
| Step 1,000 | ~0.52 Billion | 5.2876 | -- |
| Step 10,000 | ~5.24 Billion | 2.7632 | 2.6943 |
| Step 50,000 | ~26.21 Billion | 2.5275 | 2.4836 |
| Step 100,000 | ~52.43 Billion | 2.4324 | 2.3965 |
| Step 150,000 | ~78.64 Billion | 2.3355 | 2.3054 |
| Step 190,735 (Final) | 100.0 Billion | 2.2901 | 2.2690 |
📊 Zero-Shot Downstream Benchmarks
| Task / Benchmark | Metric | Score |
|---|
| HellaSwag | Acc (Norm) | 43.69% |
| ARC Easy | Acc (Norm) | 53.58% |
| ARC Challenge | Acc (Norm) | 28.92% |
| WinoGrande | Accuracy | 51.30% |
| MMLU (Overall 57-Subject Average) | Accuracy | 24.70% |
| ** ├─ MMLU STEM** | Accuracy | 23.53% |
| ** ├─ MMLU Humanities** | Accuracy | 24.87% |
| ** ├─ MMLU Social Sciences** | Accuracy | 24.86% |
| ** └─ MMLU Other** | Accuracy | 25.46% |
⚡ Long-Context Inference Hardware Scaling (32,768 Context)
BareTorch replaces context-dependent KV-caches with constant-sized $O(1)$ recurrent state updates, eliminating memory bus bottlenecks and out-of-memory crashes on long-context workloads.
1. Discrete CUDA GPU (NVIDIA RTX 4090 - 24GB)
| Baseline Pair | Context | Prefill Latency | Local GPU Decode | Peak VRAM | Advantage |
|---|
| Qwen3 0.6B Match | 32,768 | 314.89 ms vs 4,343.26 ms | 164.49 tok/s vs 13.15 tok/s | 1.66 GB vs 8.58 GB | 12.51x Faster (-80.6% VRAM) |
| SmolLM2 1.7B Match | 32,768 | 1,134.20 ms vs 4,292.54 ms | 98.92 tok/s vs 15.79 tok/s | 4.39 GB vs 15.57 GB | 6.26x Faster (-71.8% VRAM) |
| Llama 3.2 1B Match | 32,768 | 598.36 ms vs 2,960.03 ms | 169.24 tok/s vs 24.44 tok/s | 3.02 GB vs 4.67 GB | 6.92x Faster (-35.4% VRAM) |
| Gemma 2 2B Match | 32,768 | 1,049.72 ms (Baseline: 💥 OOM) | 101.51 tok/s (Baseline: 💥 OOM) | 5.96 GB (Baseline: 💥 OOM) | Prevents OOM Crashes |
2. Apple Silicon Unified Memory (M1 MacBook Pro 16GB - Native MLX)
| Baseline Pair | Context | Prefill Latency | Local GPU Decode | Peak VRAM | Advantage |
|---|
| SmolLM2 1.7B Match | 32,768 | 29.56 s vs 59.03 s | 29.69 tok/s vs 0.66 tok/s | 3.78 GB vs 9.97 GB | 44.98x Faster (-62.1% VRAM) |
| Llama 3.2 1B Match | 32,768 | 18.69 s vs 41.06 s | 28.42 tok/s vs 3.29 tok/s | 2.81 GB vs 3.71 GB | 8.64x Faster (-24.4% VRAM) |
| Qwen3 0.6B Match | 32,768 | 7.06 s (Baseline: 💥 OOM) | 79.55 tok/s (Baseline: 💥 OOM) | 1.37 GB (Baseline: 💥 OOM) | Prevents OOM Crashes |
💻 Usage & Code Example
import torch
from transformers import AutoTokenizer
from baretorch.integration.configuration_baretorch import BareTorchConfig
from baretorch.integration.modeling_baretorch import BareTorchForCausalLM
model_id = "model-rampage/BareTorch-500M-Base"
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-360M")
model = BareTorchForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()
prompt = "The key innovation of pure GEMM sub-quadratic architectures is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
📜 Citation
@article{kovacevic2026baretorch,
title={BareTorch: Challenging State-of-The-Art Sequence Mixing Topologies via Kernel-Free, Pure GEMM-Compliant Architectures},
author={Kovacevic Buvinic, Martin Ignacio},
journal={BareTorch Framework Laboratory Technical Report},
year={2026}
}