bge-reranker-v2-m3
This repository contains
BAAI/bge-reranker-v2-m3
together with a Furiosa Executable Bundle (FXB) for running it on
FuriosaAI RNGD with
Furiosa-LLM.
The same model also runs on other frameworks (such as FlagEmbedding, Sentence
Transformers, and Transformers); for usage with those, see the upstream
BAAI/bge-reranker-v2-m3
model card.
Overview
BGE Reranker v2 M3 is a multilingual cross-encoder reranking model developed by
BAAI on the XLM-RoBERTa encoder architecture. Given a query and a set of
candidate documents, it produces relevance scores used to reorder retrieval
results in retrieval-augmented generation (RAG) and search pipelines. Its
intended use is the same as the upstream
BAAI/bge-reranker-v2-m3,
and it is released under the
Apache License 2.0.
- Architecture: XLM-RoBERTa (dense encoder),
XLMRobertaForSequenceClassification - Input / Output: Text (query-document pairs) / Relevance score
- Supported Inference Engine: Furiosa LLM
- Supported Hardware: FuriosaAI RNGD
Quantization
No lower-bit quantization is applied. This repository retains the upstream FP32 checkpoint weights; Furiosa-LLM downcasts them to BF16 for W16A16 execution by the FXB.
Parallelism Strategy
On RNGD, bge-reranker-v2-m3 runs with a tensor-parallel size of 8 PEs, which maps to a single RNGD card (8 PEs per card).
Usage
To run this model with Furiosa-LLM, follow the examples below after installing Furiosa-LLM and its prerequisites. You can use the model either online through the OpenAI-compatible server or offline through the Furiosa-LLM Python API.
Launch the server
Serve the model by passing its furiosa-ai/<repo> identifier:
# Launch the server, listening on port 8000 by default
furiosa-llm serve furiosa-ai/bge-reranker-v2-m3
When the server is ready, you will see:
INFO: Started server process [27507]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Basic Usage
The server exposes a /v1/rerank endpoint compatible with the Cohere and Jina
rerank APIs. Send a query and candidate documents with curl; the server
returns the documents ordered by their sigmoid-normalized relevance_score:
curl http://localhost:8000/v1/rerank \
-H "Content-Type: application/json" \
-d '{
"model": "furiosa-ai/bge-reranker-v2-m3",
"query": "What is deep learning?",
"documents": [
"Deep learning is a subset of machine learning using neural networks.",
"Python is a popular programming language for data science.",
"Neural networks are inspired by biological neural networks."
]
}' \
| python -m json.tool
You can pass top_n to keep only the most relevant documents. To score
query-document pairs directly instead of reranking, the server also exposes a
/v1/score endpoint.
Advanced Usage
For offline use, load the model with the LLM constructor (the FXB shipped in
the repo is discovered automatically) and call score with a query and the
candidate documents to obtain relevance scores:
from furiosa_llm import LLM
query = "What is deep learning?"
documents = [
"Deep learning is a subset of machine learning using neural networks.",
"Python is a popular programming language for data science.",
]
with LLM("furiosa-ai/bge-reranker-v2-m3") as llm:
outputs = llm.score(query, documents)
for document, output in zip(documents, outputs, strict=True):
print(f"score={output.outputs.score:.4f} {document}")
Learn more
- Furiosa-LLM Server (
furiosa-llm serve) — full OpenAI-compatible API reference, including the Rerank and Score APIs - Furiosa-LLM — Furiosa-LLM documentation and API reference
BAAI/bge-reranker-v2-m3— upstream model card