OpenVision2 · L/14 @224 — vision encoder + caption decoder
This repo now contains the full OpenVision2 generative model: the vision encoder (originally released here) plus the caption text decoder it was jointly trained with. Together they map an image -> a descriptive caption. The encoder files are unchanged; only the decoder (and this card) were added.
Files
| file | role |
|---|---|
open_clip_pytorch_model.bin, open_clip_config.json | L/14 vision encoder (open_clip format) |
caption_decoder.safetensors | caption text decoder weights |
text_decoder_config.json | decoder architecture config |
modeling_openvision2_decoder.py | standalone PyTorch decoder + generate() |
bert_base_vocab_bos_eos.txt | tokenizer vocab ([PAD]=0 [bos]=1 [eos]=2) |
caption_example.py | end-to-end image -> caption demo |
Decoder architecture
A concat / prefix-LM autoregressive transformer (not a CoCa cross-attention decoder): ViT patch tokens are linearly projected and prepended as a bidirectional prefix, and text is generated causally while attending to all image tokens.
12 layers · width 768 · 12 heads · mlp 3072 · vocab 32000 · pre-LN · gelu(tanh) · LayerNorm eps 1e-6 · no positional embedding on the text stream.
It consumes the encoder's pre-final-norm patch tokens (open_clip output_tokens=True).
Usage
Needs the patched open_clip providing create_vision_encoder_and_transforms
(https://github.com/UCSC-VLAA/OpenVision), plus torch, safetensors, pillow.
See caption_example.py:
python caption_example.py --image your.jpg
Notes
- Encoder and decoder are a matched pair exported from the same training checkpoint.
- Captions are LLaVA-style dense descriptions (multi-sentence, detailed).