LLaVA-llama32-3b-VLfull-s42-s2a — VL-cap
VL-cap arm of a controlled VL-vs-text training grid built on
LLaVA-NeXT. Arm definition: S1 -> S2a, stop. Molmo-after-pretraining analog.
Backbone llama32-3b (meta-llama/Llama-3.2-3B-Instruct), seed 42, output of S2a (caption stage).
This is a vision-language arm: it trains the full VL stack on top of the stage-1 projector, with images at anyres_max_9 through google/siglip-so400m-patch14-384.
What the grid controls
Every arm sees the same mixtures, the same example order for a given seed, the same optimizer settings and the same restarted LR schedule. Arms differ only in data, order, and which modules are trainable — so differences between arms are attributable, not confounded.
Because VLfull runs its stages sequentially with the seed inherited, the S2a
checkpoint is the cap-only arm at that seed. Both stages are published separately
for exactly that reason.
Training
| Stage | S2a (caption stage) |
| Initialized from | meta-llama/Llama-3.2-3B-Instruct |
| Data mixture | CAP-750K — captions_700k_v2 (700,000) + language_50k_v2 (49,956) |
| Trainable | projector + vision tower + LM (mm_vision_tower,mm_mlp_adapter,mm_language_model) |
| Steps | 5859 / 5859 (epoch 1.0000 of 1) |
| Global batch | 128 |
| LR | 1e-5 (LM / projector), 2e-06 (vision tower), cosine, warmup ratio 0.03 |
| Precision | bfloat16 |
| Max seq length | 8192 |
| Hardware | 4x H100 80GB (DeepSpeed ZeRO-3) |
Training loss 1.497 -> 1.045
(mean of last 50 logged steps: 1.040), 0 non-finite losses across
5859 logged steps. The run reached a full epoch; trainer_state.json is included
in this repo with the complete per-step loss / grad-norm / LR history.
Usage
These weights use the LlavaLlamaForCausalLM class from the LLaVA-NeXT repo, which is
not part of transformers. Load them with that repo checked out:
from llava.model.builder import load_pretrained_model
tokenizer, model, image_processor, ctx_len = load_pretrained_model(
"BAJUKA/LLaVA-llama32-3b-VLfull-s42-s2a", None, "llava_llama"
)
Prompt template: llama_v3.
Caveats
- Research artifact from a controlled comparison, not a tuned or safety-aligned release. It inherits the licence and the limitations of meta-llama/Llama-3.2-3B-Instruct.
- The intermediate (s2a) checkpoint is a legitimate arm of the grid, not a "best" checkpoint — no early stopping or checkpoint selection was performed. Each stage simply runs one full epoch.
- Optimizer / DeepSpeed / RNG state is not published; these are inference weights.