WeMM-Embedding-2B-Quantized (Hybrid FP8 Attn/GDN + INT4-g16 MLP)

Model Details

• Model Name: WeMM-Embedding-2B-Quantized

• Developer / Publisher: ewin-reg

• Base Architecture: tencent/WeMM-Embedding-2B (2.72B total parameters, Qwen3.5 hybrid architecture)

• Model Type: Omni-modal Foundation Embedding Model (Text, Image, Video)

• Quantization Scheme: Hybrid Curvature-Guided Mixed-Precision (Per-Token FP8 E4M3 Vocab + PAS-Guarded FP8 E4M3 Attention + Group-16 Symmetric INT4 MLPs)

• Format: Single Unified SafeTensors (model.safetensors, 1,791.14 MB / 1.749 GB)

• Embedding Dimensions: 2048 native (with Matryoshka Representation Learning down to 64 dims)

• Compatibility: 100% native Hugging Face and SentenceTransformers (trust_remote_code=True)

Intended Uses & Deployment Scope

Primary Use Cases

• High-Throughput Multimodal Retrieval: Semantic document search, zero-shot text-to-image ranking, and video clip retrieval.

• Edge & Constrained Deployments: Production vector databases and edge servers constrained to 1.5 GB – 2.0 GB memory budgets.

• Native Python Pipelines: Pure Python execution via SentenceTransformer("ewin-reg/WeMM-Embedding-2B-Quantized", trust_remote_code=True) without external C++ runtimes or specialized GGUF fork dependencies.

• Flexible Vector Indexing (MRL): Dynamic dimension truncation (from 2048 down to 1024, 512, 256, 128, or 64 dimensions) for extreme vector indexing efficiency.

Out-of-Scope & Limitations

• Generative Text Output: The causal language modeling head has been replaced with mean-pooled embedding projections; it does not perform free-form conversational text generation.

• Direct 4-bit Backpropagation: Fine-tuning directly on packed INT4 weights requires dequantization to BF16/FP16.

Architecture Deep-Dive: Hybrid Attention & Quantization Mechanics

tencent/WeMM-Embedding-2B utilizes a hybrid Qwen3.5 architecture consisting of:

1. 18 Linear-Attention (GatedDeltaNet) Layers

Linear attention replaces softmax with an associative state-space recurrence:

$$S_t = \alpha_t S_{t-1} + \beta_t (v_t - S_{t-1} k_t) k_t^T$$

• Unlike full softmax attention, the recurrent state accumulates errors over time.
• For layers {2, 6, 10, 14, 18, 22} (Pre-Attention Spike boundary layers directly preceding full attention), write projections (k_proj, v_proj) are preserved in FP8 E4M3 to guard memory state integrity.
• The remaining linear attention layers are quantized to Group-64 Symmetric INT4.

2. 6 Full-Attention Layers (Layers 3, 7, 11, 15, 19, 23)

Full attention computes standard scaled dot-product attention:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d}}\right) V$$

• In standard 4-bit quantization (uniform INT4 or GGUF Q4_0), quantization noise in query Q and key K projections is amplified exponentially by softmax, causing attention collapse (D_KL >= 7.50).
• This checkpoint preserves all 6 Full-Attention layers in FP8 E4M3, eliminating exponential distortion and guaranteeing D_KL < 0.75.

3. Vocabulary Embedding Table (248,078 x 2048)

• The uncompressed BF16 embedding table requires 1,016.11 MB (over 1.0 GB alone).
• Quantized using Per-Token Scaled FP8 E4M3FN (FP8RowScaledEmbedding), reducing the table to 508.56 MB (saving 507.5 MB of disk and VRAM while preserving 99.95% token direction fidelity).

4. DeepStack Vision Transformer (24 Layers)

• 24-layer ViT processing spatial image patches (16 x 16) and temporal video frames (2 x 2).
• Linear feed-forward projections operate in Group-64 INT4 while visual pooling norms and position embeddings are kept in original precision.

Comprehensive Quantization Benchmark & Comparison

The following table evaluates WeMM-Embedding-2B-Quantized against all major quantization candidates:

Note on the image and video fidelity figures

The 94.6120 image and 93.1850 video rows in the table above were measured against an input set I no longer have, so nobody can reproduce them and I cannot defend them. A user measured the base against a quantized build on 289 photos and 244 tag queries, same inputs on both sides, and landed at 97.2912 for images at 2048 dimensions with a minimum of 92.9057. That is a different input set from mine, so the two numbers do not directly contradict each other, but theirs is documented and mine is not. Treat theirs as the figure to use, and this row as history rather than as a current claim. Their numbers live on the INT8 card with their protocol attached.

Detailed Layer-by-Layer Quantization Breakdown

Matryoshka Representation Learning (MRL) Benchmark

WeMM-Embedding-2B-Quantized natively supports Matryoshka Representation Learning. Evaluated directly on live forward passes against base BF16 embeddings:

Quickstart & Complete Inference Examples

1. Installation

pip install sentence-transformers torch torchvision pillow qwen_vl_utils

2. Multimodal Retrieval Inference (Text, Image, Video)

import torch

import torch.nn.functional as F

from sentence_transformers import SentenceTransformer

from PIL import Image

import numpy as np

# 1. Load the quantized model directly from Hugging Face Hub

model = SentenceTransformer("ewin-reg/WeMM-Embedding-2B-Quantized", trust_remote_code=True)

# 2. Encode Text Queries & Documents

texts = [

    "High-throughput vector indexing with post-training quantization.",

    "Recent advances in multimodal foundation embeddings in 2026."

]

text_embeddings = model.encode(texts)

print("Text Embeddings Shape:", text_embeddings.shape)  # (2, 2048)

# 3. Encode Images

image = Image.new("RGB", (224, 224), color=(73, 109, 137))

image_embedding = model.encode(image)

print("Image Embedding Shape:", image_embedding.shape)  # (2048,)

# 4. Matryoshka Dimension Truncation (e.g., to 1024 or 512 dimensions)

raw_vec = torch.tensor(text_embeddings)

mrl_1024 = F.normalize(raw_vec[:, :1024], p=2, dim=-1)

mrl_512 = F.normalize(raw_vec[:, :512], p=2, dim=-1)

print("Truncated MRL-1024 Shape:", mrl_1024.shape)  # (2, 1024)

print("Truncated MRL-512 Shape:", mrl_512.shape)    # (2, 512)

3. Cross-Modal Text-to-Image Ranking

import numpy as np

# Compute cosine similarity between text query and visual embedding

text_vec = text_embeddings[1] / np.linalg.norm(text_embeddings[1])

img_vec = image_embedding / np.linalg.norm(image_embedding)

similarity = float(np.dot(text_vec, img_vec))

print(f"Cross-Modal Text-to-Image Cosine Similarity: {similarity:.4f}")

4. Video Frame Sequence Embedding

# Video inputs can be processed as sequential PIL frames

frames = [Image.new("RGB", (224, 224), color=(i * 20, 100, 150)) for i in range(4)]

frame_embeddings = model.encode(frames)

# Mean-pool video temporal representations

video_embedding = np.mean(frame_embeddings, axis=0)

video_embedding = video_embedding / np.linalg.norm(video_embedding)

print("Aggregated Video Embedding Shape:", video_embedding.shape)  # (2048,)

Hardware Requirements & Performance Profiling

Notes for Apple Silicon and MPS

torch.float8_e4m3fn has no MPS kernels, so this FP8 checkpoint does not run on Apple Silicon. On a Mac, use WeMM-Embedding-2B-Apple-Silicon-MLX instead.

The INT8 Apple Silicon variant existed to cover that gap, and it has been removed. Nobody ever tested it on a Mac while it was up. A user has since run it on MPS from their own copy and posted the numbers in discussion #2: it loads under device_map="auto" onto mps:0, agrees with CPU to a worst case 1-cos of 5.12e-05, and the forced MPS path ran about 11x faster than CPU. Those are their measurements, not mine, and it is the only MPS run of that build I know of.

lm_head.weight is absent by design

config.json sets tie_word_embeddings: true, so lm_head.weight is not stored and ties to embed_tokens at load. Under device_map="auto" the meta placeholder never materializes, which is why a device dump shows lm_head on the meta device. Nothing is missing and the forward pass is unaffected.

Citation & References

@article{wemm2026,

  title={WeMM: Versatile Multimodal Foundation Embedding Model},

  author={Tencent PCG},

  journal={arXiv preprint arXiv:2608.24053},

  year={2026}

}

@inproceedings{flatquant2025,

  title={FlatQuant: Flatness-Aware Post-Training Quantization for Large Language Models},

  author={Liu, Zhen and others},

  booktitle={ICLR},

  year={2025}

}

@article{slq2026,

  title={SLQ: Statistically-Lossless Quantization of Large Language Models},

  author={Dan Alistarh and colleagues},

  journal={Conference on Language Modeling (COLM)},

  year={2026}

}