LRAT Qwen3-Embedding-0.6B — 3 Epochs
This is a full-parameter retriever checkpoint based on
Qwen/Qwen3-Embedding-0.6B, prepared for the CCIR LRAT competition.
Training
- Data: official LRAT training pairs only; no external training data
- Records per epoch: 96,504
- Cumulative epochs: 3
- Optimizer steps per epoch: 12,063
- Hardware: 2 × NVIDIA A40
- Precision: bfloat16
- Train group: 1 positive and 5 negatives
- Global effective query batch: 8
- Query / passage maximum length: 128 / 512
- Learning rate: 1e-6 for each epoch
- Pooling: last token
- Normalized embeddings: yes
- Temperature: 0.02
- Query instruction:
Given a web search query, retrieve relevant passages that answer the query
Each additional epoch starts from the preceding epoch's full model weights and
uses a freshly initialized optimizer and learning-rate schedule. Training uses
the official LRAT pair data and reweight_rate values with cross-device
negatives. The repository contains complete model parameters and does not
require a LoRA or adapter at inference time.
Loading
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
model_id = "Seraphic663/lrat-qwen3-0.6b-3epoch-20260718"
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")
model = AutoModel.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
).cuda().eval()
instruction = "Given a web search query, retrieve relevant passages that answer the query"
texts = [f"Instruct: {instruction}\nQuery:example query", "example passage"]
batch = tokenizer(
texts,
padding=True,
truncation=True,
max_length=512,
return_tensors="pt",
).to("cuda")
with torch.inference_mode():
output = model(**batch).last_hidden_state
embeddings = F.normalize(output[:, -1].float(), p=2, dim=1)
Validation Note
On a fixed 500-row candidate-set training-overlap diagnostic, this checkpoint obtained Recall@1/5/10 of 0.772/0.952/0.992 and MRR of 0.85295. All 500 queries also occur in the training pairs, so these values are regression diagnostics, not held-out results or official BC-Plus leaderboard scores.
Integrity
model.safetensors
2,383,139,480 bytes
SHA-256 996189d7ee1580cb7c9eb1b2fa550b72c1afd806f7fbdc7e6ac7305bf2e6e1b5