jina-reranker-v3.5 — GGUF

GGUF quantizations of jinaai/jina-reranker-v3.5, a domain-ready multilingual listwise reranker from Jina AI.

Available quantizations

Files

• jina-reranker-v3.5-*.gguf — Quantized model weights (see table above for sizes)
• projector.safetensors — MLP projector weights (not baked into the GGUF; required for scoring)
• tokenizer.json — Tokenizer required by rerank.py for in-process tokenization and block splitting
• rerank.py — Reranker implementation
• imatrix.dat — Importance matrix used to guide quantization of lower-bit variants

Requirements

• Python 3.8+
• llama-embedding binary — built from the fork below (see Building llama.cpp)
• projector.safetensors from this repo (the scoring MLP — not baked into the GGUF)
• Python packages: numpy, safetensors, torch, tokenizers

Building llama.cpp

This model requires a non-causal encoder mode and a custom --output-token-ids flag that are not yet in the official llama.cpp release. A PR is open upstream:

ggml-org/llama.cpp#26286

Until the PR is merged, build llama-embedding from the fork:

git clone https://github.com/littlewine/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # or -DGGML_METAL=ON on macOS
cmake --build build --config Release -j$(nproc) --target llama-embedding

The resulting binary is at build/bin/llama-embedding.

CPU-only build: omit the -DGGML_CUDA=ON flag.

Usage

from rerank import GGUFReranker

reranker = GGUFReranker(
    model_path="jina-reranker-v3.5-Q4_K_M.gguf",
    projector_path="projector.safetensors",
    llama_embedding_path="/path/to/llama-embedding",
    tokenizer_path="tokenizer.json",  # omit if tokenizer.json is next to the .gguf
)

results = reranker.rerank(
    query="What causes inflammation in rheumatoid arthritis?",
    documents=[
        "Rheumatoid arthritis is caused by immune system attacking joints...",
        "Osteoarthritis results from mechanical wear on cartilage...",
    ],
)

for r in results:
    print(f"[{r['index']}] {r['relevance_score']:.4f}  {r['document'][:60]}")

Citation

If you find jina-reranker-v3.5 useful in your research, please cite the original paper:

@misc{nasika2026jinarerankerv35efficientlistwisereranker,
      title={jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation}, 
      author={Christina Nasika and Feng Wang and Antonis Krasakis and Han Xiao},
      year={2026},
      eprint={2607.18152},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2607.18152}, 
}

License

This GGUF implementation follows the same CC BY-NC 4.0 license as the original model. For commercial usage inquiries, please contact Jina AI.