💬 BareTorch-500M-SFT
BareTorch-500M SFT is the instruction-aligned counterpart of the BareTorch-500M Base language model. It was fine-tuned on multi-turn instruction dialogue datasets using ChatML formatting across $2\times$ NVIDIA RTX 4090 GPUs.
🏋️ Supervised Fine-Tuning (SFT) Details
- Base Checkpoint: model-rampage/BareTorch-500M-Base (100B Token Pre-Trained Foundation Model)
- Dataset:
HuggingFaceTB/smol-smoltalk(~485,000 multi-turn instruction samples) - Hardware Config: $2\times$ NVIDIA GeForce RTX 4090 (24GB)
- Training Epochs: 1 Epoch ($2,759$ total steps)
- Global Batch Size: 64 sequences ($131,072$ tokens/step, sequence length 2048)
- Learning Rate: $1 \times 10^{-5}$ with 100 warmup steps
- Convergence Loss: Evaluation loss decreased smoothly from $1.5631$ (step 250) to $1.4302$ (step 2,759).
📊 Benchmark Comparison: Base vs. SFT Aligned
Instruction alignment unlocks massive leaps across reasoning, commonsense, and QA benchmarks:
| Benchmark Task | Metric | Base Model (100B) | SFT Aligned Model | Instruction Gain |
|---|---|---|---|---|
| HellaSwag | Acc-Norm | 43.69% | 52.52% | +8.83% |
| ARC Easy | Acc-Norm | 53.58% | 59.18% | +5.60% |
| ARC Challenge | Acc-Norm | 28.92% | 35.41% | +6.49% |
| WinoGrande | Acc | 51.30% | 55.80% | +4.50% |
| MMLU (Overall) | Accuracy | 24.70% | 25.57% | +0.87% |
📊 Detailed SFT Benchmark Breakdown
| Task / Benchmark | Metric | Score |
|---|---|---|
| HellaSwag | Acc (Norm) | 52.52% |
| ARC Easy | Acc (Norm) | 59.18% |
| ARC Challenge | Acc (Norm) | 35.41% |
| WinoGrande | Accuracy | 55.80% |
| MMLU (Overall 57-Subject Average) | Accuracy | 25.57% |
| ** ├─ MMLU STEM** | Accuracy | 26.51% |
| ** ├─ MMLU Humanities** | Accuracy | 25.80% |
| ** ├─ MMLU Social Sciences** | Accuracy | 23.72% |
| ** └─ MMLU Other** | Accuracy | 26.10% |
⚡ Long-Context Inference Hardware Scaling (32,768 Context)
BareTorch's kernel-free CS-LRAD hybrid engine delivers massive decoding acceleration and VRAM footprint reduction:
| Platform / Device | Context Window | Decode Velocity | Peak Memory | Speedup vs Standard Transformer |
|---|---|---|---|---|
| NVIDIA CUDA (RTX 4090) | 32,768 Tokens | 164.49 tok/s | 1,660 MB | 12.51x Faster (vs Qwen3 0.6B) |
| Apple Silicon (M1 MBP MLX) | 32,768 Tokens | 29.69 tok/s | 3,778 MB | 44.98x Faster (vs SmolLM2 1.7B) |
🤖 Prompt Format (ChatML)
<|im_start|>user
Explain quantum entanglement in two sentences.<|im_end|>
<|im_start|>assistant
Quantum entanglement is a phenomenon where two or more particles become connected such that the state of one instantly influences the state of the other, regardless of distance. Einstein famously referred to this counterintuitive phenomenon as "spooky action at a distance."<|im_end|>
💻 Code Usage Example
import torch
from transformers import AutoTokenizer
from baretorch.integration.configuration_baretorch import BareTorchConfig
from baretorch.integration.modeling_baretorch import BareTorchForCausalLM
model_id = "model-rampage/BareTorch-500M-SFT"
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-360M")
model = BareTorchForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()
messages = [
{"role": "user", "content": "What are three key benefits of pure-GEMM neural network architectures?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=150, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
📜 Citation
@article{kovacevic2026baretorch,
title={BareTorch: Challenging State-of-The-Art Sequence Mixing Topologies via Kernel-Free, Pure GEMM-Compliant Architectures},
author={Kovacevic Buvinic, Martin Ignacio},
journal={BareTorch Framework Laboratory Technical Report},
year={2026}
}