PIXEL LINGUIST II — Mid-Training Only

A vision-only encoder that reads, retrieves and compresses language directly in pixel space. Text is rendered to RGB images and encoded by the same tower that encodes natural images, so no text tokenizer is involved at inference time.

This is the "mid-training only" ablation from the paper — trained directly from the Qwen2.5-VL vision tower on the Stage 2 recipe (26M curated semantic text pairs + 26M LAION image-text pairs), skipping the Stage 1 multilingual Wikipedia pretraining. It is the checkpoint behind the paper's Table 3 and Table 4 headline rows.

Related releases:

From the EMNLP paper On the Design Fundamentals of Pixel Text Representation Learning.

Model details

Adjacent visual tokens are compressed by a 2x2 pooling layer; embeddings are mean-pooled over the (H/2)x(W/2) merged patches per image and L2-normalised.

Results

Spearman correlation for Visual STS, nDCG@5 for ViDoRe (6-subset average, the paper's convention). Qwen2.5-ViT is the untrained backbone.

This checkpoint is the paper's strongest single model on ViDoRe and already sets state of the art on English Visual STS. Adding Stage 1 pretraining on top (pixel-linguist-2-midtrain) trades a little document retrieval for markedly better cross-lingual and multilingual alignment (+3.3 cross, +3.5 multi) — the curriculum finding of the paper's RQ4.

Usage

import json, torch, torch.nn.functional as F
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoImageProcessor
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import Qwen2_5_VisionTransformerPretrainedModel
from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig

path = snapshot_download("Pixel-Linguist/Pixel-Linguist-II-Midtrain-Only")

config = Qwen2_5_VLVisionConfig(**json.load(open(f"{path}/config.json")))
model = Qwen2_5_VisionTransformerPretrainedModel(config)
model.load_state_dict(load_file(f"{path}/model.safetensors"), strict=True)
model = model.to("cuda", torch.bfloat16).eval()
processor = AutoImageProcessor.from_pretrained(path)


@torch.no_grad()
def encode(images):
    """Embed a list of PIL images (rendered text or natural photos)."""
    inputs = processor(images=images, return_tensors="pt")
    pixel_values = inputs["pixel_values"].to("cuda", torch.bfloat16)
    grid_thw = inputs["image_grid_thw"].to("cuda")

    hidden = model(hidden_states=pixel_values, grid_thw=grid_thw)

    sizes = ((grid_thw[:, 1] // 2) * (grid_thw[:, 2] // 2)).long()
    if sizes.sum() != hidden.shape[0]:
        sizes[-1] += hidden.shape[0] - sizes.sum()
    index = torch.repeat_interleave(torch.arange(len(grid_thw), device="cuda"), sizes)

    pooled = torch.zeros((len(grid_thw), hidden.shape[-1]), dtype=hidden.dtype, device="cuda")
    pooled.index_add_(0, index, hidden)
    pooled = pooled / sizes.unsqueeze(1).to(hidden.dtype).clamp(min=1)
    return F.normalize(pooled, dim=-1)

To embed a string, render it to an image first — that is the whole point of the model. Any renderer works; the paper samples from 393 fonts and 5,000+ DTD textured backgrounds during training so the encoder is robust to layout:

from PIL import Image, ImageDraw

def render(text, size=(448, 64)):
    img = Image.new("RGB", size, "white")
    ImageDraw.Draw(img).text((5, 25), text, fill="black")
    return img

emb = encode([render("a dog runs in the park"),
              render("a puppy is running outside"),
              render("quantum field theory")])
emb @ emb.T

Reproducibility notes. transformers>=4.57 loads the image processor as Qwen2VLImageProcessorFast by default and warns that this "may produce slightly different outputs"; pass use_fast=False to AutoImageProcessor.from_pretrained if you need the slow processor. Scores are also sensitive to how text is rendered — the canvas size and font size act as spatial proxies for this model, so the same string rendered at a different width will give a different embedding.

Training

Text is rendered on the fly with randomised fonts, sizes, backgrounds, brightness and blur, so the model never sees the same visual instantiation of a string twice. Natural image-text pairs act as a required regulariser — dropping them collapses document retrieval even at full data scale.

Citation

@inproceedings{yuan2026pixel,
  title     = {On the Design Fundamentals of Pixel Text Representation Learning},
  author    = {Yuan, Chaohao and Yuan, Ruifeng and Huang, Zhuoxu and Rong, Yu and
               Cheng, Hong and Chan, Hou Pong and Xiao, Chenghao},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}