A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling
C2LLMs (Code Contrastive Large Language Models) are powerful new models for generating code embeddings, designed to capture the deep semantics of source code.
Key Features
- Powerful Base Model: Built upon the state-of-the-art
Qwen2.5-Coder, inheriting its exceptional code comprehension capabilities. - Intelligent Pooling with PMA: Instead of traditional
mean poolingorlast token pooling, C2LLM uses PMA (Pooling by Multi-head Attention). This allows the model to dynamically focus on the most critical parts of the code, creating a more informative and robust embedding. - Trained for Retrieval: C2LLM was fine-tuned on a massive dataset of 3 million query-document pairs, optimizing it for real-world code retrieval and semantic search tasks. Supporting Text2Code/Code2Code/Code2Text tasks.
C2LLM is designed to be a go-to model for tasks like code search and Retrieval-Augmented Generation (RAG). For more details, please see our GitHub repository.
Model Details
How to use
Usage (HuggingFace Transformers)
from transformers import AutoModel, AutoTokenizer
import torch
model_path = "codefuse-ai/C2LLM-0.5B"
# Load the model
model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16, trust_remote_code=True)
# Prepare your custom instruction
instruction = "xxxxx"
# Prepare the data
sentences = ['''int r = (int) params >> 8 & 0xff;
int p = (int) params & 0xff;
byte[] derived1 = SCrypt.scrypt(passwd.getBytes("UTF-8"), salt, N, r, p, 32);
if (derived0.length != derived1.length) return false;
int result = 0;
for (int i = 0; i < derived0.length; i++) {
result |= derived0[i] ^ derived1[i];
}
return result == 0;
} catch (UnsupportedEncodingException e) {
throw new IllegalStateException("JVM doesn't support UTF-8?");
} catch (GeneralSecurityException e) {
throw new IllegalStateException("JVM doesn't support SHA1PRNG or HMAC_SHA256?");
}
}''',
'''
}
if (tempFrom > tempTo) {
return new RangeInfo(inclusive ? tempTo : tempTo + 1, tempFrom + 1, true);
}
return new RangeInfo(tempFrom, inclusive ? tempTo + 1 : tempTo, false);
}''']
sentences = [instruction+sentence for sentence in sentences]
# Get the embeddings
embeddings = model.encode(sentences)
Usage (Sentence-Transformers)
from sentence_transformers import SentenceTransformer
# Load the model
model = SentenceTransformer("codefuse-ai/C2LLM-0.5B", trust_remote_code=True, tokenizer_kwargs={"padding_side":"left"})
# Prepare your custom instruction
instruction = "xxxxx"
# Prepare the data
sentences = ['''int r = (int) params >> 8 & 0xff;
int p = (int) params & 0xff;
byte[] derived1 = SCrypt.scrypt(passwd.getBytes("UTF-8"), salt, N, r, p, 32);
if (derived0.length != derived1.length) return false;
int result = 0;
for (int i = 0; i < derived0.length; i++) {
result |= derived0[i] ^ derived1[i];
}
return result == 0;
} catch (UnsupportedEncodingException e) {
throw new IllegalStateException("JVM doesn't support UTF-8?");
} catch (GeneralSecurityException e) {
throw new IllegalStateException("JVM doesn't support SHA1PRNG or HMAC_SHA256?");
}
}''',
'''
}
if (tempFrom > tempTo) {
return new RangeInfo(inclusive ? tempTo : tempTo + 1, tempFrom + 1, true);
}
return new RangeInfo(tempFrom, inclusive ? tempTo + 1 : tempTo, false);
}''']
sentences = [instruction+sentence for sentence in sentences]
# Get the embeddings
embeddings = model.encode(sentences)
Evaluation (MTEB)
from sentence_transformers import SentenceTransformer
from mteb.models import ModelMeta
from mteb.cache import ResultCache
model_name = "codefuse-ai/C2LLM-0.5B"
# Load the model
model = mteb.get_model(model_name) # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)
# Select tasks
tasks = mteb.get_tasks(tasks=["AppsRetrieval", "CodeSearchNetCCRetrieval", "CodeEditSearchRetrieval","CodeSearchNetRetrieval","CodeFeedbackMT","CodeFeedbackST","CodeTransOceanContest","CodeTransOceanDL","COIRCodeSearchNetRetrieval","CosQA","StackOverflowQA","SyntheticText2SQL"])
# Cache the result
cache = ResultCache("./c2llm_results")
# Evaluate
results = mteb.evaluate(model, tasks=tasks, cache=cache, encode_kwargs={"batch_size": 16})
Support Us
If you find this project helpful, please give it a star. It means a lot to us!
Citation
@article{2025C2LLM, title={C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling}, author={Jin Qin and Zihan Liao and Ziyin Zhang and Hang Yu and Peng Di and Rui Wang}, journal = {CoRR}, volume = {abs/2512.21332}, year = {2025}, url = {https://doi.org/10.48550/arXiv.2512.21332}, doi = {10.48550/ARXIV.2512.21332}, eprinttype = {arXiv}, eprint = {2512.21332} }