Encoder-Free Vision-Language Model (VLM)
This repository hosts an Encoder-Free Vision-Language Model (VLM) wrapped around the base language model SupraLabs/Supra-1.5-50M-Base-exp and distilled using embeddings generated by SigLIP-2.
By using robust mathematical hook patches instead of standard heavy vision encoders, it extracts and maps visual tokens directly into target hidden representation vectors in the browser or terminal.
File Registry
vlm_model.onnx: Optimized FP32 ONNX model compatible with CPU/WASM onnxruntime-web execution providers.vlm_model_fp16.onnx: Optimized FP16 ONNX model for WebGPU/WebGL rendering acceleration.model.safetensors: Standard PyTorch model state dictionary (SafeTensors format).modeling_vlm.py: Custom Python wrapper code for theEncoderFreeVLMmodule andVLMPreprocessor.config.json: Hardware parameter settings.
PyTorch Integration
To load this model natively in Python, clone this repository and use the custom classes in modeling_vlm.py:
import torch
from modeling_vlm import EncoderFreeVLM, VLMPreprocessor
from transformers import AutoTokenizer, AutoModelForCausalLM
# 1. Load base components
tokenizer = AutoTokenizer.from_pretrained("SupraLabs/Supra-1.5-50M-Base-exp", trust_remote_code=True)
cma_model = AutoModelForCausalLM.from_pretrained("SupraLabs/Supra-1.5-50M-Base-exp", trust_remote_code=True)
# 2. Instantiate wrapper
preprocessor = VLMPreprocessor(tokenizer)
vlm = EncoderFreeVLM(cma_model, embedding_dim=768, proj_mode="linear")
# 3. Load model state dictionary and automatically reconstruct shared weights
from safetensors.torch import load_model
load_model(vlm, "model.safetensors")
vlm.eval()
# Example forward execution
# inputs = preprocessor(image=your_image_object)
# embeddings = vlm(**inputs)
ONNX Runtime Configuration
When deploying inside JS/Web applications, query the model with the following inputs:
input_ids[int64,[batch_size, sequence_length]]attention_mask[int64,[batch_size, sequence_length]]images[float32,[batch_size, 3, 224, 224]]is_image[float32,[batch_size, 1]]