gemma-4-31B-caveman
Gemma 4 31B fine-tuned to speak caveman-mode natively. Full bf16 weights, drop-in replacement for google/gemma-4-31B-it.
Drops articles, filler, pleasantries, hedging. Allows fragments. Keeps code blocks, function names, error strings, and CLI commands byte-exact. Pattern: [thing] [action] [reason]. [next step].
For the lighter LoRA-adapter variant see JBrussee/gemma-4-31B-caveman-lora.
Use
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("JBrussee/gemma-4-31B-caveman")
model = AutoModelForCausalLM.from_pretrained(
"JBrussee/gemma-4-31B-caveman",
torch_dtype=torch.bfloat16,
device_map="auto",
)
msgs = [{"role": "user", "content": "Explain database connection pooling."}]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(model.device)
out = model.generate(ids, max_new_tokens=300, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
Example
Prompt: "Why does my React component re-render every time the parent updates?"
Output:
Parent re-render → child re-render by default. Props "change" each render
if inline obj/array/fn → new ref → child sees new props.
Fix:
- Wrap child: React.memo(Child)
- Stabilize props: useMemo for obj/array, useCallback for fn
- Lift state so parent re-render not needed
Check: if child memo'd but still re-renders, inline prop is culprit.
Training
- Method: QLoRA NF4 + double-quant + bf16 compute (Unsloth), then merged to bf16
- LoRA: rank 16, α 32, dropout 0, targets
q,k,v,o,gate,up,down - Loss: assistant-only loss masking via TRL
completion_only_loss=True - Epochs: 3, lr 2e-4 cosine, batch 2 × grad-accum 8 (effective 16)
- Hardware: RunPod RTX PRO 6000 Blackwell 96 GB
- Time: ~50 minutes total
Data
1750 train + 193 eval pairs, source-normal → caveman style transfer, drawn from six permissive datasets:
| Source | License | Used |
|---|---|---|
| OpenAssistant/oasst2 | Apache 2.0 | dialogue |
| princeton-nlp/SWE-bench_Verified | research-permissive | debug |
| ronantakizawa/github-codereview | permissive subset | code review |
| bigcode/commitpackft (MIT/Apache subset) | MIT/Apache 2.0 | refactor |
| theblackcat102/evol-codealpaca-v1 | Apache 2.0 | short Q&A |
| HuggingFaceH4/ultrachat_200k | MIT | short Q&A |
Caveman side synthesized via Claude Code (claude -p) and Codex CLI (codex exec with GPT-5.5), routed through the SKILL.md ruleset.
Eval (n=193 holdout)
| Category | n | compression | article density | code_fence_match | semantic_sim |
|---|---|---|---|---|---|
| dialogue | 28 | 0.59 | 0.020 | 1.000 | 0.91 |
| debug | 34 | 0.92 | 0.009 | 0.995 | 0.98 |
| refactor | 27 | 0.92 | 0.005 | 0.963 | 0.98 |
| qa | 104 | 0.65 | 0.007 | 1.000 | 0.92 |
Final train_loss 0.024 · eval_loss 0.72 · eval_mean_token_accuracy 0.815.
Strengths: code preservation (96-100% fence-exact), low article density (0.5-2%), strong semantic preservation (91-98%). Weakness: compression ~10-40% rather than the 50-70% of the gold caveman pairs. To compress harder, retrain on a tighter-filtered dataset (filter upper bound 0.7 instead of 1.0).
Reproduce
Full code, data pipeline, and configs: https://github.com/JuliusBrussee/finetune-caveman
License
Inherits the Gemma Prohibited Use Policy. Apache 2.0 base + Gemma terms apply to all outputs. Repository code is MIT. The caveman style ruleset is MIT (https://github.com/JuliusBrussee/caveman).
Citing
@misc{brussee2026cavemanGemma,
author = {Julius Brussee},
title = {Caveman-mode Gemma 4 31B},
year = {2026},
url = {https://huggingface.co/JBrussee/gemma-4-31B-caveman}
}