A.X VE — Vision Encoder

Model Summary

This is the vision encoder — a Vision Transformer that encodes images into patch-level features.

This vision encoder is trained from scratch and generatively: a frozen in-house LLM decoder consumes the visual tokens and is optimized with a Next-Token-Prediction (NTP) objective, so the encoder is aligned from the very beginning to the exact form it will take in the final VLM. A.X VE is developed as part of the Korean government's Sovereign AI foundation model project.

The full A.X VLM built on top of this vision encoder will be released soon.

Key Features

• Trained from scratch, generatively. A dedicated in-house LLM decoder is used directly as the decoder, and only the generative (NTP) signal is used as the training objective. The pretraining pipeline mirrors real VLM usage, so the encoder is aligned to its final deployment form from the start.
• Two-stage curriculum (low-res alignment → high-res refinement). Stage 1 builds vision–language grounding on large-scale captioning at lower resolution; Stage 2 scales up the resolution and strengthens document/text (OCR) understanding.
• Task-diverse spatial & OCR strength. Beyond captions, training mixes Parsing / Transcription / Grounding data to directly boost document/text recognition and object-location/spatial understanding from the pretraining stage.
• Multi-level residual aggregation. Intermediate features from blocks are added back into the final hidden state, giving a multi-scale representation that combines low-level visual cues with deep semantics.

Model Details

The Vision Encoder is a Vision Model (ViT) that encodes images into patch-level features.

Vision Model

• Architecture: Vision Transformer
• Hidden size: 1152
• Intermediate size: 4304
• Number of layers: 27
• Number of attention heads: 16
• Patch size: 16 × 16
• Native resolution: supported
• Spatial merge size: 2 (2 × 2 adjacent patches merged before projection)
• Residual visual indexes: [6, 12, 18]

Weights

model.safetensors (413M parameters, bf16 with fp32 patch embedding): patch_embed, pos_embed, blocks.0 … blocks.26 (27 layers), final_layernorm.

Training

The vision encoder is trained from scratch using a generative pipeline built around an in-house LLM decoder.

Setup

• Vision Model: ViT
• Projector (training only): a 2-layer MLP that concatenates 2×2 spatially-merged patches and projects visual features into the decoder's embedding space.
• Decoder (training only, frozen): a compact in-house language model, pretrained from scratch and used only as the decoder during vision-encoder training.

The Vision Model and Projector are trainable; the LLM is frozen so its language ability is preserved and the visual features are aligned into a space the decoder can already understand. The projector aligns visual features into the language embedding space, and training focuses on vision–language alignment via Next-Token-Prediction (NTP).

Two-Stage Curriculum

The Vision Encoder is trained in two stages that differ in resolution and data composition, with the language decoder kept frozen throughout.

• Stage 1 — low-resolution alignment. Learns the basic correspondence between visual information and language from scratch, using large-scale captioning data at lower resolution. A relatively large learning rate with generous warmup and cosine decay aligns the representation space quickly and stably, emphasizing general visual and spatial understanding.
• Stage 2 — high-resolution refinement. Scales up the resolution and reinforces text/document recognition. The learning rate and warmup are lowered to minimize disruption to the Stage-1 representation, while adding Parsing / Transcription / Grounding data on top of captioning to strengthen OCR and grounding.

Training characteristics

1. Generative training with an in-house LLM decoder — the decoder is used directly and only the generative loss serves as the training signal. Because the pretraining pipeline matches how the encoder is used in the final VLM, the encoder is aligned to its final deployment form from the start, keeping the training and inference objectives consistent.
2. NTP-based learning — visual tokens from the Vision Encoder are fed to the frozen decoder, which is trained to generate text about the image; only the Vision Encoder is updated, so the visual representation is shaped into a space the decoder can consume while the LLM's language ability is preserved.
3. Task-diverse spatial & OCR strengthening — Parsing/Transcription directly strengthen document/text recognition (OCR), and Grounding directly strengthens object-location/spatial understanding, raising document understanding and position-based task performance from the pretraining stage.

Performance

Benchmark comparison. Scores for A.X VE and the SigLIP2:

Note: These results were obtained from experiments using a small in-house LLM as the decoder, so the absolute scores are for relative comparison between vision encoders rather than final VLM performance.

Usage

Requirements

pip install "transformers>=4.55.4" torch pillow

The model ships with custom modeling code, so load it with trust_remote_code=True.

Extracting image features

This checkpoint holds the vision encoder only (no language backbone), so the vision tower is instantiated directly from the config and its weights are loaded from model.safetensors.

import torch
from PIL import Image
from safetensors.torch import load_file
from transformers import AutoConfig, AutoImageProcessor
from transformers.dynamic_module_utils import get_class_from_dynamic_module

model_path = "."  # this directory (vision encoder only)

# Image processor (returns `pixel_values_images` and `image_grid_hw`)
processor = AutoImageProcessor.from_pretrained(model_path, trust_remote_code=True)

# Build the vision tower from config, then load weights
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
VisionModel = get_class_from_dynamic_module(
    "modeling_ax_ve.AXVEVisionModel", model_path
)
model = VisionModel._from_config(
    config.vision_config
).to(torch.bfloat16).eval()
model.load_state_dict(load_file(f"{model_path}/model.safetensors"), strict=True)

image = Image.open("example.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    # forward(hidden_states=pixel_values, grid_hw=image_grid_hw) -> (num_patches, 1152)
    features = model(
        inputs["pixel_values_images"].to(torch.bfloat16),
        grid_hw=inputs["image_grid_hw"],
    )

Limitations

• This is a vision encoder — it produces visual features, not text. It is meant to be used as the visual front-end of a VLM together with a language backbone, not as a standalone model.
• The output visual features are aligned to the training decoder's embedding space; pairing the encoder with a different language backbone may require additional adaptation.
• Encoding quality may vary with image domain and characteristics (resolution, aspect ratio, visual style) depending on the coverage of the training data.

License

Released under the Apache 2.0 license.

Citation

@article{AdotXVE,
  title={A.X VE},
  author={SKT A.X Team},
  year={2026},
  url={https://huggingface.co/skt/A.X-VE}
}

Contact

For questions about A.X VE — including model behavior, deployment, and licensing — contact the A.X team at a.x@sk.com. Please send reports of vulnerabilities, harmful outputs, suspected misuse, or copyright infringement claims to the same address.