Qwen text adapter for LingBot-Vision-Giant

This is a small experimental adapter model that aligns Qwen3.5-0.8B text features with frozen LingBot-Vision-Giant patch features. It is mostly just an experiment to add some of Qwen's text understanding to LingBot via training a simple text-queryable tag head on top of it.

Model

LingBot-G supplies the image representation and Qwen supplies one cached 1024-dimensional vector for each tag. A small trainable head projects both into a normalized 256-dimensional space, where a learned scale is applied to their cosine similarity to give a response for every image patch and tag.

IMAGE PATH
image
  -> frozen LingBot-G
  -> 1536D patch features
  -> learned visual projection
  -> normalized 256D patch vectors

TEXT PATH (precomputed once for the bundled vocabulary)
tag text
  -> frozen Qwen3.5-0.8B
  -> cached 1024D tag vector
  -> learned text projection
  -> normalized 256D tag vector

COMPARISON AND POOLING
patch vectors + tag vector
  -> cosine similarity x learned scale
  -> one response per patch/tag pair
  -> adaptive top-patch pooling at 1%, 3%, and 8% (minimum 2 patches)
  -> image/tag score

Only the ~1.58M-parameter alignment head is trained. For the bundled vocabulary, Qwen is run once to encode the tags. Those cached features are stored with the adapter, so Qwen is not required at inference time unless new text queries need to be encoded.

Training

The adapter was trained with image-level supervision from Booru-Essence and took about 2 hours on an rtx 5090:

• 39,576 training images and 2,107 validation images
• 2,048 training concepts
• 384 x 384 letterboxed inputs and batch size 16
• random horizontal flips with 50% probability
• binary cross-entropy with per-tag positive weights of sqrt(negative examples / positive examples), clamped to 1-20
• 16 epochs

Quick start

pip install -r requirements.txt
python inference.py assets/example.png --query sword --output sword-map.png

The first run downloads LingBot-Vision-Giant. The script prints the highest-ranked bundled concepts and optionally writes a patch-response overlay for --query. If the query is outside the bundled vocabulary, Qwen3.5-0.8B is loaded to encode it.

The example ranks snow, fox, weapon, and sword as its top four bundled concepts.

Python API

import torch

from adapter import load_adapter, top_queries

device = "cuda"

adapter, query_embeddings, config = load_adapter(".", device=device)
queries = config["tags"]

# Produce these with LingBot-Vision-Giant.
# patch_tokens: [batch, patches, 1536]
# valid_patches: [batch, patches], True for real image patches
patch_tokens = ...
valid_patches = ...

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    image_logits, patch_logits = adapter(
        patch_tokens,
        valid_patches,
        query_embeddings,
    )

print(top_queries(image_logits, queries, limit=20)[0])

patch_logits keeps the native LingBot patch order and can be reshaped to its spatial grid. adapter.py also provides encode_queries for giving Qwen arbitrary text queries, although the bundled config.json contains the concepts that it was trained on.

top_queries returns ordinal rank and query text. Spatial behavior varies with the image, query, and input resolution.

Files

• model.safetensors: the learned alignment-head weights and 2,048 cached Qwen query vectors
• config.json: architecture, training configuration, and supported query names
• adapter.py: loader, optional Qwen query encoder, and scoring implementation
• inference.py: end-to-end image ranking and response-map command-line tool
• assets/: metadata-free example image and its sword response overlay

License

The adapter is released under CC BY-NC-SA 4.0 because Booru-Essence was used for training.