LLaVA-llama32-3b-Tfull-s1337-s2a — T-cap
T-cap arm of a controlled VL-vs-text training grid built on
LLaVA-NeXT. Arm definition: stripped S2a. Text twin of VL-cap.
Backbone llama32-3b (meta-llama/Llama-3.2-3B-Instruct), seed 1337, output of S2a (caption stage).
This is a text twin: the exact same examples in the exact same order, with <image>/<video> tokens stripped and the image field removed, so only the language model is tuned. Vision-tower and projector weights are carried in the checkpoint at their initial values and are not used. It exists to isolate what VL training changes relative to consuming the same text.
What the grid controls
Every arm sees the same mixtures, the same example order for a given seed, the same optimizer settings and the same restarted LR schedule. Arms differ only in data, order, and which modules are trainable — so differences between arms are attributable, not confounded.
Because Tfull runs its stages sequentially with the seed inherited, the S2a
checkpoint is the cap-only arm at that seed. Both stages are published separately
for exactly that reason.
Training
| Stage | S2a (caption stage) |
| Initialized from | meta-llama/Llama-3.2-3B-Instruct |
| Data mixture | CAP-750K — captions_700k_v2 (700,000) + language_50k_v2 (49,956) |
| Trainable | LM only (mm_language_model); vision tower and projector are untouched |
| Steps | 5859 / 5859 (epoch 1.0000 of 1) |
| Global batch | 128 |
| LR | 1e-5 (LM / projector), cosine, warmup ratio 0.03 |
| Precision | bfloat16 |
| Max seq length | 8192 |
| Hardware | 4x H100 80GB (DeepSpeed ZeRO-3) |
Training loss 2.669 -> 1.312
(mean of last 50 logged steps: 1.338), 0 non-finite losses across
5859 logged steps. The run reached a full epoch; trainer_state.json is included
in this repo with the complete per-step loss / grad-norm / LR history.
Usage
These weights use the LlavaLlamaForCausalLM class from the LLaVA-NeXT repo, which is
not part of transformers. Load them with that repo checked out:
from llava.model.builder import load_pretrained_model
tokenizer, model, image_processor, ctx_len = load_pretrained_model(
"BAJUKA/LLaVA-llama32-3b-Tfull-s1337-s2a", None, "llava_llama"
)
Prompt template: llama_v3.
Caveats
- Research artifact from a controlled comparison, not a tuned or safety-aligned release. It inherits the licence and the limitations of meta-llama/Llama-3.2-3B-Instruct.
- The intermediate (s2a) checkpoint is a legitimate arm of the grid, not a "best" checkpoint — no early stopping or checkpoint selection was performed. Each stage simply runs one full epoch.
- Optimizer / DeepSpeed / RNG state is not published; these are inference weights.