👁️ Keural Vision Encoder (PoC · V0.1)

A 24.7M-parameter vision encoder trained from scratch — no pretrained backbone, no CLIP weights. Structured tokens that plug into any LLM.

The encoder (CNN Stem → ATB → Spatial Transformer) feeds the LevelAware Projector → Mistral-7B in the full Keural VLM.

✨ Key Innovations

🎯 Adaptive Token Budget (ATB) Tokenization — token count is a runtime parameter. Tokens are allocated to information-dense regions: a blank wall gets fewer, a dense document gets more.

out = encoder(image, token_budget=64)    # cheap
out = encoder(image, token_budget=256)   # default
out = encoder(image, token_budget=1024)  # full fidelity

🪜 Hierarchical Concept Tokenization (HCT) — every token carries a semantic level tag: global (whole-image), region (object-scale), or detail (fine-grained).

out = encoder(image)
print(out.level_ids)  # tensor of {0=global, 1=region, 2=detail}

⚙️ Model Specifications

🎨 Saliency & Token Placement

The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement (global / region / detail).

🧪 Training

Phase 1 — Vision Encoder Pretraining ✅ COMPLETE. Trained from scratch for ~75,000 steps on CC3M + CC12M (~6.9M image–text pairs), SigLIP-style contrastive objective, 1× RTX 5090. The frozen encoder is then integrated into the full VLM via LevelAwareProjector (384 → 2048 → 4096) → Mistral-7B-Instruct-v0.3 (4-bit NF4 QLoRA), fine-tuned with SFT (LLaVA-Instruct-150K, 30K steps).

📊 Benchmark Results — SFT-30K

Downstream VLM benchmarks using this encoder (frozen) + projector + Mistral-7B at the SFT-30K checkpoint (supervised fine-tuning, 30,000 steps; before DPO). Evaluated on 1,000 samples each where applicable. The Keural encoder is 12.4× smaller than LLaVA's CLIP encoder (307M).

These are the SFT-30K numbers (measured from outputs/eval/sft_30k/). Applying DPO alignment (RLHF-V, 3K steps) improves every metric — e.g. VQAv2 → 43.6%, ScienceQA → 53.7%, MME → 838.8. Full SFT+DPO results, the comparison chart, and the complete VLM live on mkd-hika/keural-vlm-poc.

⚡ Adaptive Token Budget — latency vs budget

Encode latency vs token budget — trade visual detail for speed at runtime, no retraining.

📐 Output Contract

Every forward pass returns a KeuralEncoderOutput:

🚀 Usage

import torch
from architecture.encoder import KeuralVisionEncoder
from keural_config import KeuralConfig

cfg = KeuralConfig.from_yaml("configs/keural_tiny_poc.yaml")
model = KeuralVisionEncoder.from_pretrained("mkd-hika/keural-vision-encoder-poc")
model.eval()

from PIL import Image
from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((256, 256)),
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5]),
])

image = Image.open("image.jpg").convert("RGB")
x = transform(image).unsqueeze(0)  # (1, 3, 256, 256)

with torch.no_grad():
    out = model(x)

print(out.tokens.shape)        # (1, 256, 384)
print(out.level_ids[0, :8])    # [0, 0, 1, 1, 1, 2, 2, 2]
print(out.pooled.shape)        # (1, 384)

🗺️ Roadmap

The mid-level model will use knowledge distillation from SigLIP-400M (natural images) and InternViT-300M (documents/OCR). Teachers are discarded after training — not part of the final model.

📝 Citation

@misc{keural_vision_encoder_2026,
  title  = {Keural Vision Encoder: Content-Adaptive Vision Encoding via Saliency-Guided Token Budgets},
  author = {Barki, Hika and MKD Co., Ltd.},
  year   = {2026},
  note   = {V0.1 — Phase 1 complete},
}

📄 License

Code: MIT. Model weights trained on CC3M + CC12M — data licenses apply.

MKD Co., Ltd. — 2026