Encoder-Free Vision-Language Model (VLM)

This repository hosts an Encoder-Free Vision-Language Model (VLM) wrapped around the base language model SupraLabs/SupraCMA-8M and distilled using embeddings generated by SigLIP-2.

By using robust mathematical hook patches instead of standard heavy vision encoders, it extracts and maps visual tokens directly into target hidden representation vectors in the browser or terminal.

File Registry

• vlm_model.onnx: Optimized FP32 ONNX model compatible with CPU/WASM onnxruntime-web execution providers.
• vlm_model_fp16.onnx: Optimized FP16 ONNX model for WebGPU/WebGL rendering acceleration.
• model.safetensors: Standard PyTorch model state dictionary (SafeTensors format).
• modeling_vlm.py: Custom Python wrapper code for the EncoderFreeVLM module and VLMPreprocessor.
• config.json: Hardware parameter settings.

PyTorch Integration

To load this model natively in Python, clone this repository and use the custom classes in modeling_vlm.py:

import torch
from modeling_vlm import EncoderFreeVLM, VLMPreprocessor
from transformers import AutoTokenizer, AutoModelForCausalLM

# 1. Load base components
tokenizer = AutoTokenizer.from_pretrained("SupraLabs/SupraCMA-8M", trust_remote_code=True)
cma_model = AutoModelForCausalLM.from_pretrained("SupraLabs/SupraCMA-8M", trust_remote_code=True)

# 2. Instantiate wrapper
preprocessor = VLMPreprocessor(tokenizer)
vlm = EncoderFreeVLM(cma_model, embedding_dim=768, proj_mode="linear")

# 3. Load model state dictionary and automatically reconstruct shared weights
from safetensors.torch import load_model
load_model(vlm, "model.safetensors")
vlm.eval()

# Example forward execution
# inputs = preprocessor(image=your_image_object)
# embeddings = vlm(**inputs)

ONNX Runtime Configuration

When deploying inside JS/Web applications, query the model with the following inputs:

• input_ids [int64, [batch_size, sequence_length]]
• attention_mask [int64, [batch_size, sequence_length]]
• images [float32, [batch_size, 3, 224, 224]]
• is_image [float32, [batch_size, 1]]

Web Deployment (JavaScript / ONNX Runtime Web)

Because image scaling, division, and mean/standard deviation normalization are baked directly into the model's ONNX computation graph, you do not need to do complex mathematical normalization in JavaScript. You only need to extract the raw [0, 255] pixel values.

1. Installation / Setup

Include the ONNX Runtime Web script in your project:

<!-- Load the latest ONNX Runtime Web CDN -->
<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.min.js"></script>

2. Client-Side Image Preprocessing

This helper function extracts raw pixel data from an HTML5 <canvas> element and structures it into the expected flat, channel-first (NCHW) format:

/**
 * Preprocesses a 224x224 canvas to a raw NCHW float32 array in the [0, 255] range.
 * @param {HTMLCanvasElement} canvas - Canvas element resized to 224x224.
 * @returns {ort.Tensor} ONNX Tensor of shape [1, 3, 224, 224]
 */
function preprocessCanvas(canvas) {
    const ctx = canvas.getContext('2d');
    const imgData = ctx.getImageData(0, 0, 224, 224).data;
    
    const floatData = new Float32Array(3 * 224 * 224);
    const numPixels = 224 * 224;
    
    for (let i = 0; i < numPixels; i++) {
        floatData[i] = imgData[i * 4];                 // Red channel
        floatData[i + numPixels] = imgData[i * 4 + 1];     // Green channel
        floatData[i + 2 * numPixels] = imgData[i * 4 + 2]; // Blue channel (Alpha is ignored)
    }
    
    return new ort.Tensor('float32', floatData, [1, 3, 224, 224]);
}

3. Running Inference

To run the unified model, construct the correct tensor layout depending on whether you are querying a text or an image:

// Initialize the ONNX session
const session = await ort.InferenceSession.create('./vlm_model_fp16.onnx', {
    executionProviders: ['webgpu', 'wasm'] // Falls back to WASM if WebGPU is unavailable
});

/**
 * Generate a 768-dimensional embedding vector for an image
 */
async function getImageEmbedding(canvas) {
    const imagesTensor = preprocessCanvas(canvas);
    
    // Provide minimal dummy values for the text inputs
    const inputIdsTensor = new ort.Tensor('int64', new BigInt64Array([0n]), [1, 1]);
    const attentionMaskTensor = new ort.Tensor('int64', new BigInt64Array([1n]), [1, 1]);
    const isImageTensor = new ort.Tensor('float32', new Float32Array([1.0]), [1, 1]); // 1.0 flags image path

    const feeds = {
        input_ids: inputIdsTensor,
        attention_mask: attentionMaskTensor,
        images: imagesTensor,
        is_image: isImageTensor
    };

    const results = await session.run(feeds);
    return results.embeddings.data; // Float32Array [768]
}

/**
 * Generate a 768-dimensional embedding vector for text
 * @param {Array<number>} tokenIds - Tokenized integer IDs (generated by your JS tokenizer)
 * @param {Array<number>} attentionMask - Token attention mask (usually all 1s)
 */
async function getTextEmbedding(tokenIds, attentionMask) {
    const seqLen = tokenIds.length;
    
    const inputIdsTensor = new ort.Tensor('int64', new BigInt64Array(tokenIds.map(BigInt)), [1, seqLen]);
    const attentionMaskTensor = new ort.Tensor('int64', new BigInt64Array(attentionMask.map(BigInt)), [1, seqLen]);
    
    // Provide a zeroed-out dummy tensor for the image inputs
    const imagesTensor = new ort.Tensor('float32', new Float32Array(3 * 224 * 224), [1, 3, 224, 224]);
    const isImageTensor = new ort.Tensor('float32', new Float32Array([0.0]), [1, 1]); // 0.0 flags text path

    const feeds = {
        input_ids: inputIdsTensor,
        attention_mask: attentionMaskTensor,
        images: imagesTensor,
        is_image: isImageTensor
    };

    const results = await session.run(feeds);
    return results.embeddings.data; // Float32Array [768]
}