KaLM-Embedding-V2 is a versatile and compact embedding model, which achieves impressive performance in general-purpose text embedding tasks by leveraging superior training techniques and data.
KaLM-embedding-multilingual-mini-instruct-v2 is trained from Qwen/Qwen2-0.5B with massive weakly-supervised pre-training and high-quality supervised fine-tuning data.
The model incorporates several innovative designs:
- Architectural Design: integration of bidirectional attention, enhancing representation learning.
- Training Recipe: multi-stage training strategy, progressively improving the generalization and performance.
- Training Objective: focal-style reweighting mechanism and online hard-negative mixing strategy to improve the efficiency and continuity of embedding training.
- Training Data: 20 categories of data for pre-training and 100 categories of data for fine-tuning, as well as comprehensive recipes for curating training datasets.
Model Information
- Model Size: 0.5B
- Embedding Dimension: 896
- Max Input Tokens: 32k
- MRL: 896 512 256 128 64
📑 Open-source Plan
- Model Checkpoint
- Training and Evaluation Code: HITsz-TMG/KaLM-Embedding
- Technical Report: KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
- Pre-training Data: Pre-training Data
- Fine-tuning Data: Fine-tuning Data
Evaluation
Overall results on MTEB (cmn, v1) and MTEB (eng, v1).
Detailed model performance on MTEB (cmn, v1).
Detailed model performance on MTEB (eng, v1).
Requirements
Since we have used the Qwen2 model, we advise you to install transformers>=4.37.0, or you might encounter the following error:
KeyError: 'qwen2'
Usage
sentence-transformers support
Using this model becomes easy when you have sentence-transformers installed:
pip install -U sentence-transformers
Then you can use the model like this:
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer("{MODEL_NAME_OR_PATH}", trust_remote_code=True, truncate_dim=None, model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "flash_attention_2"})
model.max_seq_length = 512
embeddings = model.encode(
sentences,
normalize_embeddings=True,
batch_size=256,
show_progress_bar=True
)
print(embeddings)
We add task instructions for queries in asymmetric tasks: retrieval, reranking, classification, and clustering.
And, we add task instructions for both queries and passages in symmetric tasks: STS and pair classification.
If you want to add task instructions to the query, you can use the model like this:
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer("{MODEL_NAME_OR_PATH}", trust_remote_code=True, truncate_dim=None, model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "flash_attention_2"})
model.max_seq_length = 512
prompt = "Instruct: Classifying the category of french news. \n Query: "
embeddings = model.encode(
sentences,
prompt=prompt,
normalize_embeddings=True,
batch_size=256,
show_progress_bar=True
)
print(embeddings)
vllm support
pip install -U vllm==0.8.5
import torch
import vllm
from vllm import LLM
def get_detailed_instruct(task_description: str, query: str) -> str:
return f'Instruct: {task_description}\nQuery:{query}'
task = 'Given a query, retrieve documents that answer the query'
queries = [
get_detailed_instruct(task, 'What is the capital of China?'),
get_detailed_instruct(task, 'Explain gravity')
]
documents = [
"The capital of China is Beijing.",
"Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun."
]
input_texts = queries + documents
model = LLM(model="{MODEL_NAME_OR_PATH}", task="embed", trust_remote_code=True, dtype="float16")
outputs = model.embed(input_texts)
embeddings = torch.tensor([o.outputs.embedding for o in outputs])
scores = (embeddings[:2] @ embeddings[2:].T)
print(scores.tolist())
Citation
If you find this model useful, please consider giving a star and citation.
@misc{zhao2025kalmembeddingv2,
title={KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model},
author={Xinping Zhao and Xinshuo Hu and Zifei Shan and Shouzheng Huang and Yao Zhou and Xin Zhang and Zetian Sun and Zhenyu Liu and Dongfang Li and Xinyuan Wei and Youcheng Pan and Yang Xiang and Meishan Zhang and Haofen Wang and Jun Yu and Baotian Hu and Min Zhang},
year={2025},
eprint={2506.20923},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2506.20923},
}
@misc{hu2025kalmembedding,
title={KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model},
author={Xinshuo Hu and Zifei Shan and Xinping Zhao and Zetian Sun and Zhenyu Liu and Dongfang Li and Shaolin Ye and Xinyuan Wei and Qian Chen and Baotian Hu and Haofen Wang and Jun Yu and Min Zhang},
year={2025},
eprint={2501.01028},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.01028},
}
Contact
If you encounter any issue, feel free to contact us via the email: zhaoxinping@stu.hit.edu.cn, yanshek.woo@gmail.com